Jul 13 21:29:17.646955 ip-10-0-140-193 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.030868    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036411    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036422    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036427    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036430    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036433    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036436    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036439    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:29:18.123064 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036443    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:29:18.102838 ip-10-0-140-193 systemd[1]: Started Kubernetes Kubelet.
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036445    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036448    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036450    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036453    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036455    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036458    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036460    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036463    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036465    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036468    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036470    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036473    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036475    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036477    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036480    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036482    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036485    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036487    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036490    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036492    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:29:18.125585 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036495    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036497    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036500    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036502    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036504    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036509    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036513    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036516    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036519    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036522    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036527    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036529    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036532    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036535    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036537    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036539    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036542    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036545    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036548    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:29:18.126989 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036550    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036552    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036555    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036557    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036560    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036562    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036564    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036567    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036570    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036572    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036574    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036577    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036579    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036582    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036584    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036587    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036589    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036591    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036594    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036597    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:29:18.128056 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036599    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036602    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036604    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036607    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036610    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036613    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036615    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036618    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036620    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036623    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036625    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036627    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036630    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036632    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036635    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036637    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036639    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036642    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036644    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:29:18.129479 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036646    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036650    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036654    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036656    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036659    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036663    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036666    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036668    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036671    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036673    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036692    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036696    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.036699    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037359    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037366    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037370    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037374    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037376    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037385    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:29:18.130593 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037387    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037390    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037393    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037395    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037398    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037401    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037403    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037405    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037408    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037412    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037415    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037418    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037421    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037424    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037427    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037429    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037431    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037434    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037437    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:29:18.131620 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037439    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037441    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037444    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037447    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037449    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037452    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037454    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037457    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037459    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037462    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037464    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037466    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037469    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037477    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037479    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037482    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037487    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037490    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037492    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037495    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:29:18.132659 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037497    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037500    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037502    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037504    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037507    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037512    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037515    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037518    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037521    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037524    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037527    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037530    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037533    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037535    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037537    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037540    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037542    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037545    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037547    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:29:18.133751 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037550    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037552    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037554    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037557    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037559    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037561    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037564    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037573    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037575    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037579    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037582    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037585    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037589    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037591    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037594    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037596    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037598    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037601    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037603    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037606    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:29:18.134553 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037608    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037610    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037614    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037616    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037619    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037621    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037623    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037626    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037629    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037631    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037633    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037636    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037638    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037649    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.037652    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038115    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038120    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038123    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038128    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:29:18.135318 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038132    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038135    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038140    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038144    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038147    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038151    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038154    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038156    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038159    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038161    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038164    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038167    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038169    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038172    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038175    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038177    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038179    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038182    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038184    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038187    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:29:18.135930 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038190    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038192    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038195    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038197    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038200    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038202    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038205    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038207    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038210    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038212    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038215    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038217    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038220    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038223    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038226    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038231    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038233    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038236    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038238    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038241    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:29:18.136517 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038243    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038246    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038248    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038250    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038253    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038256    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038258    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038260    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038263    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038265    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038268    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038270    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038273    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038275    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038277    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038280    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038282    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038285    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038287    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:29:18.137215 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038290    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038294    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038296    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038299    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038301    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038303    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038306    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038310    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038313    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038316    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038320    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038323    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038326    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038329    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038332    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038335    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038338    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038340    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038343    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:29:18.138044 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038346    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038350    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038352    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038354    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038357    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038360    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038362    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038364    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038367    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038369    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038372    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038375    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038377    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038379    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038383    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038386    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.038388    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038456    2582 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038463    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038479    2582 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038487    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 21:29:18.138939 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038494    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038499    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038503    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038507    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038510    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038513    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038516    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038520    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038523    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038526    2582 flags.go:64] FLAG: --cgroup-root=""
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038529    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038531    2582 flags.go:64] FLAG: --client-ca-file=""
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038534    2582 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038537    2582 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038541    2582 flags.go:64] FLAG: --cluster-domain=""
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038544    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038547    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038550    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038553    2582 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038557    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038560    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038563    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038566    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038569    2582 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 21:29:18.139928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038572    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038576    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038579    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038582    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038586    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038589    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038592    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038596    2582 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038599    2582 flags.go:64] FLAG: --enable-server="true"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038602    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038606    2582 flags.go:64] FLAG: --event-burst="100"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038609    2582 flags.go:64] FLAG: --event-qps="50"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038612    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038615    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038618    2582 flags.go:64] FLAG: --eviction-hard=""
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038622    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038625    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038628    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038631    2582 flags.go:64] FLAG: --eviction-soft=""
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038634    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038637    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038640    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038642    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038645    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038648    2582 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 21:29:18.140824 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038651    2582 flags.go:64] FLAG: --feature-gates=""
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038654    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038657    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038661    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038663    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038666    2582 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038669    2582 flags.go:64] FLAG: --help="false"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038672    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-140-193.ec2.internal"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038690    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038693    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038696    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038699    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038702    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038705    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038711    2582 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038714    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038717    2582 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038720    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038723    2582 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038726    2582 flags.go:64] FLAG: --kube-reserved=""
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038729    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038732    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038735    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038738    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 21:29:18.142091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038741    2582 flags.go:64] FLAG: --lock-file=""
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038743    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038747    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038749    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038755    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038758    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038760    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038763    2582 flags.go:64] FLAG: --logging-format="text"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038766    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038769    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038772    2582 flags.go:64] FLAG: --manifest-url=""
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038775    2582 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038779    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038782    2582 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038786    2582 flags.go:64] FLAG: --max-pods="110"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038789    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038793    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038796    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038799    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038802    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038805    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038808    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038816    2582 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038819    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038822    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 21:29:18.143258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038825    2582 flags.go:64] FLAG: --pod-cidr=""
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038829    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038835    2582 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038837    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038840    2582 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038843    2582 flags.go:64] FLAG: --port="10250"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038846    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038849    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0154be3df9aa8abc8"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038852    2582 flags.go:64] FLAG: --qos-reserved=""
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038856    2582 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038859    2582 flags.go:64] FLAG: --register-node="true"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038862    2582 flags.go:64] FLAG: --register-with-taints=""
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038865    2582 flags.go:64] FLAG: --registry-burst="10"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038868    2582 flags.go:64] FLAG: --registry-qps="5"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038871    2582 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038874    2582 flags.go:64] FLAG: --reserved-memory=""
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038877    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038880    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038883    2582 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038886    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038889    2582 flags.go:64] FLAG: --runonce="false"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038904    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038908    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038912    2582 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038914    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 21:29:18.144957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038917    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038920    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038923    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038926    2582 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038928    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038932    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038935    2582 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038938    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038940    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038944    2582 flags.go:64] FLAG: --system-cgroups=""
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038946    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038951    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038954    2582 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038957    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038979    2582 flags.go:64] FLAG: --tls-min-version=""
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038982    2582 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038985    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038988    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038991    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038994    2582 flags.go:64] FLAG: --v="2"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.038998    2582 flags.go:64] FLAG: --version="false"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.039002    2582 flags.go:64] FLAG: --vmodule=""
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.039005    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.039008    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 21:29:18.146370 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039132    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039136    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039139    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039142    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039145    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039148    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039152    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039155    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039157    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039160    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039163    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039166    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039169    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039173    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039176    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039178    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039181    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039184    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039187    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039190    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:29:18.327116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039193    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039196    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039199    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039202    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039204    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039207    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039210    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039212    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039215    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039217    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039222    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039225    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039228    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039230    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039233    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039237    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039240    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039243    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039247    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039250    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:29:18.343149 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039253    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039256    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039258    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039261    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039264    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039269    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039271    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039274    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039276    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039279    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039281    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039284    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039288    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039291    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039294    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039296    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039299    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039302    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039305    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039307    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:29:18.343796 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039310    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039313    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039316    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039318    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039321    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039324    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039326    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039329    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039331    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039334    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039340    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039342    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039345    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039347    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039350    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039352    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039355    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039360    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039362    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039365    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:29:18.344404 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039367    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039370    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039372    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039375    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039378    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039380    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039383    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039385    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039388    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039390    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039393    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039395    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039398    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039400    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039403    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039405    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039408    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039410    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:29:18.344972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.039413    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.040007    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.047085    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.047102    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.047762    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.047772    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047841    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047847    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047850    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047853    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047856    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047859    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047862    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047864    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047867    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:29:18.345476 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047869    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047872    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047874    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047877    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047880    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047882    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047886    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047888    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047891    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047895    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047899    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047902    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047904    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047907    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047909    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047912    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047915    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047917    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047920    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047923    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:29:18.345955 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047931    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047934    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047936    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047939    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047941    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047944    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047949    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047951    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047954    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047956    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047959    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047961    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047963    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047966    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047968    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047971    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047973    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047976    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047978    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047981    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:29:18.346529 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047984    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047987    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047990    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047992    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047995    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.047997    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048000    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048002    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048005    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048007    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048010    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048013    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048017    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048025    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048028    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048031    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048034    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048036    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048041    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:29:18.347099 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048043    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048046    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048048    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048051    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048053    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048056    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048058    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048061    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048063    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048065    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048068    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048070    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048073    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048075    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048078    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048080    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048084    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048087    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048089    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048092    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048094    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:29:18.347641 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048097    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048099    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048102    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048104    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048107    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048109    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048117    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048119    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048122    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048126    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.048132    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048281    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048285    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048288    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 21:29:18.348213 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048291    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048294    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048297    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048299    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048302    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048304    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048307    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048313    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048317    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048320    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048324    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048327    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048329    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048332    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048337    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048339    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048342    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048345    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048347    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 21:29:18.348640 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048350    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048352    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048355    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048357    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048360    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048362    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048403    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048511    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.048516    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049224    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049234    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049239    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049244    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049250    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049254    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049259    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049263    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049268    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049272    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 21:29:18.349156 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049276    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049287    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049292    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049296    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049300    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049305    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049309    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049313    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049318    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049321    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049326    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049329    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049333    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049337    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049346    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049350    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049354    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049358    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049363    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049367    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049371    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 21:29:18.349739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049374    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049379    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049387    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049392    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049395    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049403    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049407    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049411    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049416    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049419    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049423    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049427    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049431    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049435    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049439    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049445    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049449    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049453    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049633    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 21:29:18.350301 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049638    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049648    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049652    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049657    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049661    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049665    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049669    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049672    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049716    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049721    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049724    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049729    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049738    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049742    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049746    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049752    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049757    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 21:29:18.350914 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:18.049762    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.049770    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.049993    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.052651    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.054171    2582 server.go:1019] "Starting client certificate rotation"
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.054269    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.054308    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.056821    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.058305    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.068667    2582 log.go:25] "Validated CRI v1 runtime API"
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.071268    2582 log.go:25] "Validated CRI v1 image API"
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.072867    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 21:29:18.351356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.075008    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c2388f4e-09b5-4bb7-8d6a-1d18269713b0:/dev/nvme0n1p3 fb0b88d8-a30f-42a2-a96f-0f47855acd47:/dev/nvme0n1p4]
Jul 13 21:29:18.351747 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.075027    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 21:29:18.351819 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.079809    2582 manager.go:216] Machine: {Timestamp:2026-07-13 21:29:18.078633364 +0000 UTC m=+0.310499866 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3174138 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22338ac1fdc60d6df8760c268d557b SystemUUID:ec22338a-c1fd-c60d-6df8-760c268d557b BootID:f783a8d1-8e1d-43f3-b996-9c805685fb77 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:46:63:21:47:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:46:63:21:47:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:e4:9a:be:98:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 21:29:18.351819 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.079899    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 21:29:18.351819 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.079966    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 21:29:18.351819 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.082983    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 21:29:18.351819 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.083007    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-193.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 21:29:18.351819 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.083635    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 21:29:18.351819 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.083781    2582 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.083798    2582 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.083806    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.084326    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.086057    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.089047    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.089159    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.089199    2582 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.089223    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.089240    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.089249    2582 kubelet.go:398] "Adding apiserver pod source"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.089258    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.089656    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.089710    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.093985    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.096628    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.096656    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.096849    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.096861    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.096867    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.096872    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.096879    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.096883    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.097253    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.097259    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.097266    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.097272    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.097279    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.097286    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.098317    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.098409    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.099161    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.099171    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.101895    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-193.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.102017    2582 server.go:1268] "Started kubelet"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.102103    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.102178    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.102191    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.103528    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.103830    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kvwx2"
Jul 13 21:29:18.352134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.104433    2582 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.107122    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.112108    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.112114    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kvwx2"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.112593    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.112900    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.113424    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.113437    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.113438    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.113585    2582 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.113595    2582 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.113623    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-193.ec2.internal.18c1f6aed0b99615  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-193.ec2.internal,UID:ip-10-0-140-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-193.ec2.internal,},FirstTimestamp:2026-07-13 21:29:18.101992981 +0000 UTC m=+0.333859483,LastTimestamp:2026-07-13 21:29:18.101992981 +0000 UTC m=+0.333859483,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-193.ec2.internal,}"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.116248    2582 factory.go:55] Registering systemd factory
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.116259    2582 factory.go:223] Registration of the systemd container factory successfully
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.116427    2582 factory.go:153] Registering CRI-O factory
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.116435    2582 factory.go:223] Registration of the crio container factory successfully
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.116471    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.116499    2582 factory.go:103] Registering Raw factory
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.116512    2582 manager.go:1196] Started watching for new ooms in manager
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.117052    2582 manager.go:349] Starting recovery of all containers
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.121981    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.137201    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-193.ec2.internal\" not found" node="ip-10-0-140-193.ec2.internal"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.138697    2582 manager.go:354] Recovery completed
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.145065    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.146948    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.146968    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.146978    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.147468    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.147478    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.147477    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.147517    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.148157    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.148189    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.148203    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.148595    2582 policy_none.go:49] "None policy: Start"
Jul 13 21:29:18.353483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.148608    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.148617    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.149463    2582 policy_none.go:47] "Start"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.193754    2582 manager.go:337] "Starting Device Plugin manager"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.193930    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.193942    2582 server.go:87] "Starting device plugin registration server"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.194111    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.194123    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.194418    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.194488    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.194495    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.194870    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.194919    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.280636    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.281785    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.281815    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.281832    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.281861    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.283828    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.294483    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.295150    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.295168    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.295177    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.295196    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-193.ec2.internal"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.301435    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-193.ec2.internal"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.301450    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-193.ec2.internal\": node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:18.355065 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.314800    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.382180    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal"]
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.382248    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.383705    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.383730    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.383740    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.384739    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.384862    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.384895    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.385360    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.385382    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.385396    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.385431    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.385449    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.385458    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.391621    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.391643    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.392254    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.392276    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.392285    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeHasSufficientPID"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.404269    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-193.ec2.internal\" not found" node="ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.409884    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-193.ec2.internal\" not found" node="ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.414029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9034b8f06e010d2f76d7e8c886303374-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal\" (UID: \"9034b8f06e010d2f76d7e8c886303374\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.414049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9034b8f06e010d2f76d7e8c886303374-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal\" (UID: \"9034b8f06e010d2f76d7e8c886303374\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:18.414066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7766334a1c1b5307e0876e60a26fe5a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-193.ec2.internal\" (UID: \"f7766334a1c1b5307e0876e60a26fe5a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.609470 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:18.414978    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:19.610525 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:19.589635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9034b8f06e010d2f76d7e8c886303374-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal\" (UID: \"9034b8f06e010d2f76d7e8c886303374\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.610525 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:19.589657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9034b8f06e010d2f76d7e8c886303374-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal\" (UID: \"9034b8f06e010d2f76d7e8c886303374\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.610525 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:19.589672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7766334a1c1b5307e0876e60a26fe5a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-193.ec2.internal\" (UID: \"f7766334a1c1b5307e0876e60a26fe5a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.610525 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:19.589716    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7766334a1c1b5307e0876e60a26fe5a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-193.ec2.internal\" (UID: \"f7766334a1c1b5307e0876e60a26fe5a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.610525 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:19.589722    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9034b8f06e010d2f76d7e8c886303374-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal\" (UID: \"9034b8f06e010d2f76d7e8c886303374\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.610525 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:19.589770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9034b8f06e010d2f76d7e8c886303374-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal\" (UID: \"9034b8f06e010d2f76d7e8c886303374\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.610525 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:19.590682    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:19.769327 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:19.691276    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:19.944129 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:19.782496    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.944129 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:19.789211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:19.944129 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:19.792307    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:19.944129 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:19.893036    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:20.119515 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:19.993433    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:20.119515 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:20.093824    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:20.283719 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.128965    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 21:29:20.283719 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.129090    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 21:29:20.283719 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.129134    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 21:29:20.283719 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:20.193879    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:20.283719 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.194866    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 21:24:19 +0000 UTC" deadline="2028-04-05 18:16:47.166392794 +0000 UTC"
Jul 13 21:29:20.283719 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.194891    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15164h47m26.971505125s"
Jul 13 21:29:20.283719 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.204067    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:29:20.440221 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:20.294216    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-193.ec2.internal\" not found"
Jul 13 21:29:20.440221 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.315701    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:29:20.440221 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.389140    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:20.440221 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.399620    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 21:29:20.440221 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.400495    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal"
Jul 13 21:29:20.440221 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.406324    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 21:29:20.440221 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:20.434956    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9034b8f06e010d2f76d7e8c886303374.slice/crio-bd32301c79f4402746b4d89bbfff47d9b666dcc72f2f891bfb047610bce2a3ce WatchSource:0}: Error finding container bd32301c79f4402746b4d89bbfff47d9b666dcc72f2f891bfb047610bce2a3ce: Status 404 returned error can't find the container with id bd32301c79f4402746b4d89bbfff47d9b666dcc72f2f891bfb047610bce2a3ce
Jul 13 21:29:20.440221 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:20.435339    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7766334a1c1b5307e0876e60a26fe5a.slice/crio-d21f6ec91e315a4e345d5e9f76ae5e8fec8db94bf6537ec1d3960eed29c54699 WatchSource:0}: Error finding container d21f6ec91e315a4e345d5e9f76ae5e8fec8db94bf6537ec1d3960eed29c54699: Status 404 returned error can't find the container with id d21f6ec91e315a4e345d5e9f76ae5e8fec8db94bf6537ec1d3960eed29c54699
Jul 13 21:29:20.441989 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:20.441481    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:29:21.165893 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.165634    2582 apiserver.go:52] "Watching apiserver"
Jul 13 21:29:21.173972 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.173462    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 21:29:21.175386 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.175358    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-pmd7k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq","openshift-cluster-node-tuning-operator/tuned-8hglh","openshift-dns/node-resolver-rh74n","openshift-multus/multus-additional-cni-plugins-wf9hl","openshift-network-diagnostics/network-check-target-bkp9m","openshift-ovn-kubernetes/ovnkube-node-vm77k","kube-system/global-pull-secret-syncer-x6tw2","kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal","openshift-image-registry/node-ca-tbqlb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal","openshift-multus/multus-778w5","openshift-multus/network-metrics-daemon-2njdj","openshift-network-operator/iptables-alerter-kdnjq"]
Jul 13 21:29:21.177356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.177264    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.178510 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.178239    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:21.178510 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.178316    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:21.179439 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.179414    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.180371 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.179897    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p2qx7\""
Jul 13 21:29:21.180371 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.179980    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 21:29:21.180371 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.180226    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 21:29:21.180371 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.180263    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 21:29:21.180625 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.180529    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 21:29:21.181048 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.180708    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 21:29:21.181048 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.180755    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.181592 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.181419    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 21:29:21.182398 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.181994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 21:29:21.182398 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.182294    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.182538 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.182460    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5bhx2\""
Jul 13 21:29:21.183513 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.182632    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 21:29:21.183513 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.183215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:21.183513 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.183268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:21.183513 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.183397    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 21:29:21.183513 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.183411    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 21:29:21.183513 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.183411    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sjfbl\""
Jul 13 21:29:21.185581 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.184176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.185581 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.185238    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 21:29:21.185581 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.185422    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 21:29:21.185581 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.185509    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 21:29:21.185581 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.185560    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 21:29:21.185859 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.185609    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 21:29:21.185859 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.185565    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8pp7f\""
Jul 13 21:29:21.185859 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.185779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:21.186817 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.186786    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 21:29:21.186893 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.186825    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 21:29:21.187167 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.187033    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5gb6l\""
Jul 13 21:29:21.187167 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.187077    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.189494 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.188220    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 21:29:21.189494 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.188334    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 21:29:21.189494 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.188572    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 21:29:21.189494 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.188620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.189494 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.188758    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-llx9z\""
Jul 13 21:29:21.189494 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.189034    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 21:29:21.189494 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.189134    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 21:29:21.189494 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.189188    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 21:29:21.189494 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.189429    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:21.189494 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.189494    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:21.190136 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.189624    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-smdsh\""
Jul 13 21:29:21.190433 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.190258    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 21:29:21.191001 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.190670    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.191001 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.190824    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 21:29:21.191455 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.191216    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nn45c\""
Jul 13 21:29:21.191455 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.191320    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 21:29:21.191455 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.191217    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 21:29:21.192957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.192937    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 21:29:21.193052 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.193036    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bc9kb\""
Jul 13 21:29:21.199913 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.199896    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 21:29:21.200045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.200027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-run-netns\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.200113 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.200062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-sysconfig\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.200113 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.200086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-lib-modules\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.200235 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.200216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-host\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.200465 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.200305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7b5bb8c-8520-4b35-8657-b66b7e5be404-cni-binary-copy\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.200649 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.200629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc6v9\" (UniqueName: \"kubernetes.io/projected/dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81-kube-api-access-mc6v9\") pod \"iptables-alerter-kdnjq\" (UID: \"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81\") " pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.200904 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.200884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-system-cni-dir\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.200965 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.200957    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-cni-dir\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.201061 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.201045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5k9h\" (UniqueName: \"kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h\") pod \"network-check-target-bkp9m\" (UID: \"28c8c935-0114-47da-bcf9-4dc1c5642f3d\") " pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:21.201125 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.201073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-os-release\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.201125 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.201110    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-run-netns\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.201253 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.201190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-conf-dir\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.201335 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.201318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cht4s\" (UniqueName: \"kubernetes.io/projected/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-kube-api-access-cht4s\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.201384 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.201356    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-run-systemd\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.201441 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.201399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-var-lib-openvswitch\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.201515 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.201499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-cni-bin\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.201864 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.201847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzs84\" (UniqueName: \"kubernetes.io/projected/1da0ad36-59f6-439c-b099-dc6884ff2431-kube-api-access-jzs84\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.202112 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.202097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9dd52d08-2a36-4291-ad30-52d7bb0ba254-agent-certs\") pod \"konnectivity-agent-pmd7k\" (UID: \"9dd52d08-2a36-4291-ad30-52d7bb0ba254\") " pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:21.202298 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.202283    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9dd52d08-2a36-4291-ad30-52d7bb0ba254-konnectivity-ca\") pod \"konnectivity-agent-pmd7k\" (UID: \"9dd52d08-2a36-4291-ad30-52d7bb0ba254\") " pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:21.202456 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.202442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-run-k8s-cni-cncf-io\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.202628 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.202554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-var-lib-kubelet\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.202754 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.202741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-log-socket\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.202890 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.202876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.203043 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.203014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:21.203214 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.203199    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-daemon-config\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.203590 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.203570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-etc-kubernetes\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.203782 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.203768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.204015 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.203963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-sys\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.204190 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-tuned\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.204379 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-cnibin\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.204465 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204385    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-os-release\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.204465 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-hostroot\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.204465 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-kubelet\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.204622 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-systemd-units\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.204622 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-kubernetes\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.204622 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.204622 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0fb821cc-e52a-468e-b50b-afc5e461d6c5-ovn-node-metrics-cert\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.204622 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdx9\" (UniqueName: \"kubernetes.io/projected/0fb821cc-e52a-468e-b50b-afc5e461d6c5-kube-api-access-nmdx9\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.204839 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-run-multus-certs\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.204839 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-registration-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.204839 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-device-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.204941 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/721b2fc5-6fe6-49a2-abaf-ea836e1d93bb-host\") pod \"node-ca-tbqlb\" (UID: \"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb\") " pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.204941 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzs8\" (UniqueName: \"kubernetes.io/projected/721b2fc5-6fe6-49a2-abaf-ea836e1d93bb-kube-api-access-ljzs8\") pod \"node-ca-tbqlb\" (UID: \"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb\") " pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.204941 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7b5bb8c-8520-4b35-8657-b66b7e5be404-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.204941 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-socket-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.205060 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-sysctl-conf\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.205060 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.204977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-cnibin\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.205060 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-slash\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.205191 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205055    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-run-openvswitch\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.205191 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0fb821cc-e52a-468e-b50b-afc5e461d6c5-env-overrides\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.205191 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0fb821cc-e52a-468e-b50b-afc5e461d6c5-ovnkube-script-lib\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.205191 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-systemd\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.205376 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-kubelet-config\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:21.205376 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-var-lib-cni-multus\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.205376 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-sysctl-d\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.205376 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205303    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4473539f-ed4b-437c-9814-8ee29ac74bf6-tmp-dir\") pod \"node-resolver-rh74n\" (UID: \"4473539f-ed4b-437c-9814-8ee29ac74bf6\") " pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.205376 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/721b2fc5-6fe6-49a2-abaf-ea836e1d93bb-serviceca\") pod \"node-ca-tbqlb\" (UID: \"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb\") " pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.205376 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81-host-slash\") pod \"iptables-alerter-kdnjq\" (UID: \"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81\") " pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.205376 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205366    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-dbus\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:21.205687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-run-ovn\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.205687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205414    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f47hs\" (UniqueName: \"kubernetes.io/projected/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-kube-api-access-f47hs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:21.205687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.205687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gk5\" (UniqueName: \"kubernetes.io/projected/f59c65a1-d32f-4b56-9217-c5bab1c467a9-kube-api-access-89gk5\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.205687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205497    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81-iptables-alerter-script\") pod \"iptables-alerter-kdnjq\" (UID: \"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81\") " pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.205687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205528    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-run\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.205687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-system-cni-dir\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.205687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:21.205687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0fb821cc-e52a-468e-b50b-afc5e461d6c5-ovnkube-config\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.205687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.206182 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-modprobe-d\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.206182 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1da0ad36-59f6-439c-b099-dc6884ff2431-tmp\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.206182 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205779    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-socket-dir-parent\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.206182 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205823    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-var-lib-cni-bin\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.206182 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-var-lib-kubelet\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.206182 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.205891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-etc-openvswitch\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.206562 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.206480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7b5bb8c-8520-4b35-8657-b66b7e5be404-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.207509 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.207417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-cni-netd\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.207600 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.207534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4473539f-ed4b-437c-9814-8ee29ac74bf6-hosts-file\") pod \"node-resolver-rh74n\" (UID: \"4473539f-ed4b-437c-9814-8ee29ac74bf6\") " pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.207600 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.207569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppr6h\" (UniqueName: \"kubernetes.io/projected/4473539f-ed4b-437c-9814-8ee29ac74bf6-kube-api-access-ppr6h\") pod \"node-resolver-rh74n\" (UID: \"4473539f-ed4b-437c-9814-8ee29ac74bf6\") " pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.207600 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.207596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sddcs\" (UniqueName: \"kubernetes.io/projected/a7b5bb8c-8520-4b35-8657-b66b7e5be404-kube-api-access-sddcs\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.208617 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.208592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-cni-binary-copy\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.208706 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.208646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-node-log\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.208706 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.208683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-sys-fs\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.216868 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.216848    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2bgcf"
Jul 13 21:29:21.233922 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.233903    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2bgcf"
Jul 13 21:29:21.290006 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.289979    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 21:29:21.309800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.309776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-system-cni-dir\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.309902 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.309808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:21.309902 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.309824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0fb821cc-e52a-468e-b50b-afc5e461d6c5-ovnkube-config\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.309902 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.309842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.309902 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.309876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-modprobe-d\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.309902 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.309898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1da0ad36-59f6-439c-b099-dc6884ff2431-tmp\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.310126 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.309902    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-system-cni-dir\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.310126 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.309920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-socket-dir-parent\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.310126 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.309949    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:21.310126 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.310030    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret podName:4271181f-f9a6-4bc2-ab11-9e21b0afebd2 nodeName:}" failed. No retries permitted until 2026-07-13 21:29:21.809998045 +0000 UTC m=+2.966391877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret") pod "global-pull-secret-syncer-x6tw2" (UID: "4271181f-f9a6-4bc2-ab11-9e21b0afebd2") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:21.310126 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310053    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.310126 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-socket-dir-parent\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.310126 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-var-lib-cni-bin\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.310126 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-var-lib-kubelet\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.310126 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-etc-openvswitch\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7b5bb8c-8520-4b35-8657-b66b7e5be404-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-cni-netd\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310174    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-modprobe-d\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-var-lib-kubelet\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4473539f-ed4b-437c-9814-8ee29ac74bf6-hosts-file\") pod \"node-resolver-rh74n\" (UID: \"4473539f-ed4b-437c-9814-8ee29ac74bf6\") " pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-etc-openvswitch\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310272    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-var-lib-cni-bin\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310303    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-cni-netd\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4473539f-ed4b-437c-9814-8ee29ac74bf6-hosts-file\") pod \"node-resolver-rh74n\" (UID: \"4473539f-ed4b-437c-9814-8ee29ac74bf6\") " pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppr6h\" (UniqueName: \"kubernetes.io/projected/4473539f-ed4b-437c-9814-8ee29ac74bf6-kube-api-access-ppr6h\") pod \"node-resolver-rh74n\" (UID: \"4473539f-ed4b-437c-9814-8ee29ac74bf6\") " pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.310414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sddcs\" (UniqueName: \"kubernetes.io/projected/a7b5bb8c-8520-4b35-8657-b66b7e5be404-kube-api-access-sddcs\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-cni-binary-copy\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310447    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-node-log\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-sys-fs\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-run-netns\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-sysconfig\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-lib-modules\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-host\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310580    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310602    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7b5bb8c-8520-4b35-8657-b66b7e5be404-cni-binary-copy\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6v9\" (UniqueName: \"kubernetes.io/projected/dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81-kube-api-access-mc6v9\") pod \"iptables-alerter-kdnjq\" (UID: \"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81\") " pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-system-cni-dir\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-cni-dir\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310683    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5k9h\" (UniqueName: \"kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h\") pod \"network-check-target-bkp9m\" (UID: \"28c8c935-0114-47da-bcf9-4dc1c5642f3d\") " pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7b5bb8c-8520-4b35-8657-b66b7e5be404-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.310940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310705    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-os-release\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-run-netns\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310738    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-node-log\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310743    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-sys-fs\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-conf-dir\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-run-netns\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cht4s\" (UniqueName: \"kubernetes.io/projected/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-kube-api-access-cht4s\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310816    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-conf-dir\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-sysconfig\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-lib-modules\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-run-systemd\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-run-systemd\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310952    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-host\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310987    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-run-netns\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.310999    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0fb821cc-e52a-468e-b50b-afc5e461d6c5-ovnkube-config\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.311629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311041    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-cni-binary-copy\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-var-lib-openvswitch\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-cni-dir\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-cni-bin\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311130    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-system-cni-dir\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-os-release\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzs84\" (UniqueName: \"kubernetes.io/projected/1da0ad36-59f6-439c-b099-dc6884ff2431-kube-api-access-jzs84\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-cni-bin\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-var-lib-openvswitch\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9dd52d08-2a36-4291-ad30-52d7bb0ba254-agent-certs\") pod \"konnectivity-agent-pmd7k\" (UID: \"9dd52d08-2a36-4291-ad30-52d7bb0ba254\") " pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9dd52d08-2a36-4291-ad30-52d7bb0ba254-konnectivity-ca\") pod \"konnectivity-agent-pmd7k\" (UID: \"9dd52d08-2a36-4291-ad30-52d7bb0ba254\") " pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-run-k8s-cni-cncf-io\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-var-lib-kubelet\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-log-socket\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.312365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-run-k8s-cni-cncf-io\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-log-socket\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-daemon-config\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311464    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-var-lib-kubelet\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311464    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7b5bb8c-8520-4b35-8657-b66b7e5be404-cni-binary-copy\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311510    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.311539    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-etc-kubernetes\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.311603    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs podName:71c50b57-6bb8-4e0f-b2a9-1daab13a74cf nodeName:}" failed. No retries permitted until 2026-07-13 21:29:21.811581793 +0000 UTC m=+2.967975612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs") pod "network-metrics-daemon-2njdj" (UID: "71c50b57-6bb8-4e0f-b2a9-1daab13a74cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-etc-kubernetes\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-sys\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-tuned\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.313023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-cnibin\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-os-release\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-hostroot\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-sys\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311767    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-kubelet\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-systemd-units\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-kubernetes\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311858    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-kubelet\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-hostroot\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-os-release\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0fb821cc-e52a-468e-b50b-afc5e461d6c5-ovn-node-metrics-cert\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdx9\" (UniqueName: \"kubernetes.io/projected/0fb821cc-e52a-468e-b50b-afc5e461d6c5-kube-api-access-nmdx9\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311949    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9dd52d08-2a36-4291-ad30-52d7bb0ba254-konnectivity-ca\") pod \"konnectivity-agent-pmd7k\" (UID: \"9dd52d08-2a36-4291-ad30-52d7bb0ba254\") " pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:21.313738 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311962    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-multus-daemon-config\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-run-multus-certs\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311991    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-kubernetes\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-registration-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.311944    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-systemd-units\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312032    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-device-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-run-multus-certs\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/721b2fc5-6fe6-49a2-abaf-ea836e1d93bb-host\") pod \"node-ca-tbqlb\" (UID: \"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb\") " pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312054    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-registration-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzs8\" (UniqueName: \"kubernetes.io/projected/721b2fc5-6fe6-49a2-abaf-ea836e1d93bb-kube-api-access-ljzs8\") pod \"node-ca-tbqlb\" (UID: \"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb\") " pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-device-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7b5bb8c-8520-4b35-8657-b66b7e5be404-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-socket-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.315800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/721b2fc5-6fe6-49a2-abaf-ea836e1d93bb-host\") pod \"node-ca-tbqlb\" (UID: \"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb\") " pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-sysctl-conf\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-cnibin\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-slash\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-run-openvswitch\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0fb821cc-e52a-468e-b50b-afc5e461d6c5-env-overrides\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0fb821cc-e52a-468e-b50b-afc5e461d6c5-ovnkube-script-lib\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312340    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-host-slash\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-systemd\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312331    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-sysctl-conf\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-kubelet-config\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-var-lib-cni-multus\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-sysctl-d\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-socket-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4473539f-ed4b-437c-9814-8ee29ac74bf6-tmp-dir\") pod \"node-resolver-rh74n\" (UID: \"4473539f-ed4b-437c-9814-8ee29ac74bf6\") " pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.316409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312496    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-cnibin\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/721b2fc5-6fe6-49a2-abaf-ea836e1d93bb-serviceca\") pod \"node-ca-tbqlb\" (UID: \"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb\") " pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81-host-slash\") pod \"iptables-alerter-kdnjq\" (UID: \"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81\") " pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-host-var-lib-cni-multus\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312542    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-dbus\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-run-ovn\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f47hs\" (UniqueName: \"kubernetes.io/projected/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-kube-api-access-f47hs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312614    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-sysctl-d\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gk5\" (UniqueName: \"kubernetes.io/projected/f59c65a1-d32f-4b56-9217-c5bab1c467a9-kube-api-access-89gk5\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7b5bb8c-8520-4b35-8657-b66b7e5be404-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81-iptables-alerter-script\") pod \"iptables-alerter-kdnjq\" (UID: \"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81\") " pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-run\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-dbus\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:21.317079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312776    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-run-ovn\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-kubelet-config\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312778    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f59c65a1-d32f-4b56-9217-c5bab1c467a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-systemd\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312827    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1da0ad36-59f6-439c-b099-dc6884ff2431-run\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4473539f-ed4b-437c-9814-8ee29ac74bf6-tmp-dir\") pod \"node-resolver-rh74n\" (UID: \"4473539f-ed4b-437c-9814-8ee29ac74bf6\") " pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.312919    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fb821cc-e52a-468e-b50b-afc5e461d6c5-run-openvswitch\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.313032    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81-host-slash\") pod \"iptables-alerter-kdnjq\" (UID: \"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81\") " pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.313190    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7b5bb8c-8520-4b35-8657-b66b7e5be404-cnibin\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.313217    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/721b2fc5-6fe6-49a2-abaf-ea836e1d93bb-serviceca\") pod \"node-ca-tbqlb\" (UID: \"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb\") " pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.313315    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0fb821cc-e52a-468e-b50b-afc5e461d6c5-env-overrides\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.313507    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81-iptables-alerter-script\") pod \"iptables-alerter-kdnjq\" (UID: \"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81\") " pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.314174    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0fb821cc-e52a-468e-b50b-afc5e461d6c5-ovnkube-script-lib\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.314761    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9dd52d08-2a36-4291-ad30-52d7bb0ba254-agent-certs\") pod \"konnectivity-agent-pmd7k\" (UID: \"9dd52d08-2a36-4291-ad30-52d7bb0ba254\") " pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.315105    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1da0ad36-59f6-439c-b099-dc6884ff2431-tmp\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.317708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.316020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1da0ad36-59f6-439c-b099-dc6884ff2431-etc-tuned\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.318326 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.317585    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0fb821cc-e52a-468e-b50b-afc5e461d6c5-ovn-node-metrics-cert\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.318326 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.317595    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:29:21.318326 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.317615    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:29:21.318326 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.317628    2582 projected.go:196] Error preparing data for projected volume kube-api-access-s5k9h for pod openshift-network-diagnostics/network-check-target-bkp9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:21.318326 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.317679    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h podName:28c8c935-0114-47da-bcf9-4dc1c5642f3d nodeName:}" failed. No retries permitted until 2026-07-13 21:29:21.817664192 +0000 UTC m=+2.974058015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s5k9h" (UniqueName: "kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h") pod "network-check-target-bkp9m" (UID: "28c8c935-0114-47da-bcf9-4dc1c5642f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:21.320007 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.319980    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sddcs\" (UniqueName: \"kubernetes.io/projected/a7b5bb8c-8520-4b35-8657-b66b7e5be404-kube-api-access-sddcs\") pod \"multus-additional-cni-plugins-wf9hl\" (UID: \"a7b5bb8c-8520-4b35-8657-b66b7e5be404\") " pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.320839 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.320815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc6v9\" (UniqueName: \"kubernetes.io/projected/dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81-kube-api-access-mc6v9\") pod \"iptables-alerter-kdnjq\" (UID: \"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81\") " pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.325849 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.325826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzs8\" (UniqueName: \"kubernetes.io/projected/721b2fc5-6fe6-49a2-abaf-ea836e1d93bb-kube-api-access-ljzs8\") pod \"node-ca-tbqlb\" (UID: \"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb\") " pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.325937 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.325841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gk5\" (UniqueName: \"kubernetes.io/projected/f59c65a1-d32f-4b56-9217-c5bab1c467a9-kube-api-access-89gk5\") pod \"aws-ebs-csi-driver-node-rq6kq\" (UID: \"f59c65a1-d32f-4b56-9217-c5bab1c467a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.325937 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.325860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cht4s\" (UniqueName: \"kubernetes.io/projected/8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d-kube-api-access-cht4s\") pod \"multus-778w5\" (UID: \"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d\") " pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.326619 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.326266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdx9\" (UniqueName: \"kubernetes.io/projected/0fb821cc-e52a-468e-b50b-afc5e461d6c5-kube-api-access-nmdx9\") pod \"ovnkube-node-vm77k\" (UID: \"0fb821cc-e52a-468e-b50b-afc5e461d6c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.326619 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.326527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppr6h\" (UniqueName: \"kubernetes.io/projected/4473539f-ed4b-437c-9814-8ee29ac74bf6-kube-api-access-ppr6h\") pod \"node-resolver-rh74n\" (UID: \"4473539f-ed4b-437c-9814-8ee29ac74bf6\") " pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.326619 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.326580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzs84\" (UniqueName: \"kubernetes.io/projected/1da0ad36-59f6-439c-b099-dc6884ff2431-kube-api-access-jzs84\") pod \"tuned-8hglh\" (UID: \"1da0ad36-59f6-439c-b099-dc6884ff2431\") " pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.327762 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.327742    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f47hs\" (UniqueName: \"kubernetes.io/projected/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-kube-api-access-f47hs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:21.362418 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.362181    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal" event={"ID":"9034b8f06e010d2f76d7e8c886303374","Type":"ContainerStarted","Data":"bd32301c79f4402746b4d89bbfff47d9b666dcc72f2f891bfb047610bce2a3ce"}
Jul 13 21:29:21.363504 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.363474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal" event={"ID":"f7766334a1c1b5307e0876e60a26fe5a","Type":"ContainerStarted","Data":"d21f6ec91e315a4e345d5e9f76ae5e8fec8db94bf6537ec1d3960eed29c54699"}
Jul 13 21:29:21.418069 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.418010    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:29:21.469937 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.469907    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 21:29:21.493947 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.493922    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:21.502788 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.502762    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8hglh"
Jul 13 21:29:21.513403 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.513379    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rh74n"
Jul 13 21:29:21.520963 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.520942    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wf9hl"
Jul 13 21:29:21.530518 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.530496    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kdnjq"
Jul 13 21:29:21.537997 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.537977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:21.545417 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.545394    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq"
Jul 13 21:29:21.551820 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.551805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tbqlb"
Jul 13 21:29:21.556516 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.556501    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-778w5"
Jul 13 21:29:21.815956 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.815919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:21.816137 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.815997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:21.816137 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.816091    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:21.816246 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.816163    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret podName:4271181f-f9a6-4bc2-ab11-9e21b0afebd2 nodeName:}" failed. No retries permitted until 2026-07-13 21:29:22.816133899 +0000 UTC m=+3.972527720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret") pod "global-pull-secret-syncer-x6tw2" (UID: "4271181f-f9a6-4bc2-ab11-9e21b0afebd2") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:21.816246 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.816091    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:21.816246 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.816213    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs podName:71c50b57-6bb8-4e0f-b2a9-1daab13a74cf nodeName:}" failed. No retries permitted until 2026-07-13 21:29:22.816200781 +0000 UTC m=+3.972594601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs") pod "network-metrics-daemon-2njdj" (UID: "71c50b57-6bb8-4e0f-b2a9-1daab13a74cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:21.917010 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:21.916973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5k9h\" (UniqueName: \"kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h\") pod \"network-check-target-bkp9m\" (UID: \"28c8c935-0114-47da-bcf9-4dc1c5642f3d\") " pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:21.917211 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.917094    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:29:21.917211 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.917124    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:29:21.917211 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.917134    2582 projected.go:196] Error preparing data for projected volume kube-api-access-s5k9h for pod openshift-network-diagnostics/network-check-target-bkp9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:21.917325 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:21.917223    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h podName:28c8c935-0114-47da-bcf9-4dc1c5642f3d nodeName:}" failed. No retries permitted until 2026-07-13 21:29:22.917203125 +0000 UTC m=+4.073596961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5k9h" (UniqueName: "kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h") pod "network-check-target-bkp9m" (UID: "28c8c935-0114-47da-bcf9-4dc1c5642f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:22.237130 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.237078    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 21:24:21 +0000 UTC" deadline="2028-03-09 09:26:11.222114564 +0000 UTC"
Jul 13 21:29:22.237130 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.237123    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14507h56m48.985005409s"
Jul 13 21:29:22.303739 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:22.303632    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b5bb8c_8520_4b35_8657_b66b7e5be404.slice/crio-724c7a2878085132930cdd19ea81d6021624e97d796f0a2380dad5e6a331fe4f WatchSource:0}: Error finding container 724c7a2878085132930cdd19ea81d6021624e97d796f0a2380dad5e6a331fe4f: Status 404 returned error can't find the container with id 724c7a2878085132930cdd19ea81d6021624e97d796f0a2380dad5e6a331fe4f
Jul 13 21:29:22.304848 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:22.304819    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcedcbc0_ec41_414b_a6fa_3fa67b1f2a81.slice/crio-38b3c4769b4cc71b722db3c0ccc4df0eda79690a89270b4ab90c2c0e4c1c6351 WatchSource:0}: Error finding container 38b3c4769b4cc71b722db3c0ccc4df0eda79690a89270b4ab90c2c0e4c1c6351: Status 404 returned error can't find the container with id 38b3c4769b4cc71b722db3c0ccc4df0eda79690a89270b4ab90c2c0e4c1c6351
Jul 13 21:29:22.305600 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:22.305473    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59c65a1_d32f_4b56_9217_c5bab1c467a9.slice/crio-162167a261d5bd9d8bd3148320152eebf2378c3fdf5ba612380a50ff5ebe24a1 WatchSource:0}: Error finding container 162167a261d5bd9d8bd3148320152eebf2378c3fdf5ba612380a50ff5ebe24a1: Status 404 returned error can't find the container with id 162167a261d5bd9d8bd3148320152eebf2378c3fdf5ba612380a50ff5ebe24a1
Jul 13 21:29:22.309036 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:22.308995    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb821cc_e52a_468e_b50b_afc5e461d6c5.slice/crio-24a60ed4624cb241ce7e9daebc1e652938ca3222c1a7550279e499e70f7d0fce WatchSource:0}: Error finding container 24a60ed4624cb241ce7e9daebc1e652938ca3222c1a7550279e499e70f7d0fce: Status 404 returned error can't find the container with id 24a60ed4624cb241ce7e9daebc1e652938ca3222c1a7550279e499e70f7d0fce
Jul 13 21:29:22.310691 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:22.310670    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4473539f_ed4b_437c_9814_8ee29ac74bf6.slice/crio-60197ad16d212f818eb2328bbbfa80e4ed8c4f2684ebbd7ea1a87480a71fbb23 WatchSource:0}: Error finding container 60197ad16d212f818eb2328bbbfa80e4ed8c4f2684ebbd7ea1a87480a71fbb23: Status 404 returned error can't find the container with id 60197ad16d212f818eb2328bbbfa80e4ed8c4f2684ebbd7ea1a87480a71fbb23
Jul 13 21:29:22.311300 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:22.311279    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721b2fc5_6fe6_49a2_abaf_ea836e1d93bb.slice/crio-aefcac3b7de178e5c5c7f20ac62e603dd52776f2bd96371d1bff9d18ba869c2f WatchSource:0}: Error finding container aefcac3b7de178e5c5c7f20ac62e603dd52776f2bd96371d1bff9d18ba869c2f: Status 404 returned error can't find the container with id aefcac3b7de178e5c5c7f20ac62e603dd52776f2bd96371d1bff9d18ba869c2f
Jul 13 21:29:22.313643 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:22.313283    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fce6eb3_9db6_42b2_b5e1_0a7d1a1c4e4d.slice/crio-5af57d0fba94c756d3aef2b73d8b689fbaae59872815bb79eb8ef17295b1e735 WatchSource:0}: Error finding container 5af57d0fba94c756d3aef2b73d8b689fbaae59872815bb79eb8ef17295b1e735: Status 404 returned error can't find the container with id 5af57d0fba94c756d3aef2b73d8b689fbaae59872815bb79eb8ef17295b1e735
Jul 13 21:29:22.316202 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:22.316178    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd52d08_2a36_4291_ad30_52d7bb0ba254.slice/crio-06a2d80a3a96cbd821cd5ca545e4dbc5aaf56481056092cefbb481980e3a5da8 WatchSource:0}: Error finding container 06a2d80a3a96cbd821cd5ca545e4dbc5aaf56481056092cefbb481980e3a5da8: Status 404 returned error can't find the container with id 06a2d80a3a96cbd821cd5ca545e4dbc5aaf56481056092cefbb481980e3a5da8
Jul 13 21:29:22.317722 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:22.317696    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da0ad36_59f6_439c_b099_dc6884ff2431.slice/crio-457b1f0c5cf0ee81e59eb3f109cb00782dadc073162ba10bf400c53d270cf2c0 WatchSource:0}: Error finding container 457b1f0c5cf0ee81e59eb3f109cb00782dadc073162ba10bf400c53d270cf2c0: Status 404 returned error can't find the container with id 457b1f0c5cf0ee81e59eb3f109cb00782dadc073162ba10bf400c53d270cf2c0
Jul 13 21:29:22.366821 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.366790    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq" event={"ID":"f59c65a1-d32f-4b56-9217-c5bab1c467a9","Type":"ContainerStarted","Data":"162167a261d5bd9d8bd3148320152eebf2378c3fdf5ba612380a50ff5ebe24a1"}
Jul 13 21:29:22.367631 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.367610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kdnjq" event={"ID":"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81","Type":"ContainerStarted","Data":"38b3c4769b4cc71b722db3c0ccc4df0eda79690a89270b4ab90c2c0e4c1c6351"}
Jul 13 21:29:22.368578 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.368558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf9hl" event={"ID":"a7b5bb8c-8520-4b35-8657-b66b7e5be404","Type":"ContainerStarted","Data":"724c7a2878085132930cdd19ea81d6021624e97d796f0a2380dad5e6a331fe4f"}
Jul 13 21:29:22.369504 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.369480    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8hglh" event={"ID":"1da0ad36-59f6-439c-b099-dc6884ff2431","Type":"ContainerStarted","Data":"457b1f0c5cf0ee81e59eb3f109cb00782dadc073162ba10bf400c53d270cf2c0"}
Jul 13 21:29:22.370514 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.370488    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pmd7k" event={"ID":"9dd52d08-2a36-4291-ad30-52d7bb0ba254","Type":"ContainerStarted","Data":"06a2d80a3a96cbd821cd5ca545e4dbc5aaf56481056092cefbb481980e3a5da8"}
Jul 13 21:29:22.371563 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.371544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-778w5" event={"ID":"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d","Type":"ContainerStarted","Data":"5af57d0fba94c756d3aef2b73d8b689fbaae59872815bb79eb8ef17295b1e735"}
Jul 13 21:29:22.372532 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.372513    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" event={"ID":"0fb821cc-e52a-468e-b50b-afc5e461d6c5","Type":"ContainerStarted","Data":"24a60ed4624cb241ce7e9daebc1e652938ca3222c1a7550279e499e70f7d0fce"}
Jul 13 21:29:22.373607 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.373581    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tbqlb" event={"ID":"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb","Type":"ContainerStarted","Data":"aefcac3b7de178e5c5c7f20ac62e603dd52776f2bd96371d1bff9d18ba869c2f"}
Jul 13 21:29:22.375434 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.375415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rh74n" event={"ID":"4473539f-ed4b-437c-9814-8ee29ac74bf6","Type":"ContainerStarted","Data":"60197ad16d212f818eb2328bbbfa80e4ed8c4f2684ebbd7ea1a87480a71fbb23"}
Jul 13 21:29:22.823374 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.823283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:22.823374 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.823369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:22.823654 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:22.823636    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:22.823718 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:22.823707    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs podName:71c50b57-6bb8-4e0f-b2a9-1daab13a74cf nodeName:}" failed. No retries permitted until 2026-07-13 21:29:24.823687652 +0000 UTC m=+5.980081471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs") pod "network-metrics-daemon-2njdj" (UID: "71c50b57-6bb8-4e0f-b2a9-1daab13a74cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:22.823792 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:22.823781    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:22.823848 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:22.823818    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret podName:4271181f-f9a6-4bc2-ab11-9e21b0afebd2 nodeName:}" failed. No retries permitted until 2026-07-13 21:29:24.823807817 +0000 UTC m=+5.980201638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret") pod "global-pull-secret-syncer-x6tw2" (UID: "4271181f-f9a6-4bc2-ab11-9e21b0afebd2") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:22.924113 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:22.924041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5k9h\" (UniqueName: \"kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h\") pod \"network-check-target-bkp9m\" (UID: \"28c8c935-0114-47da-bcf9-4dc1c5642f3d\") " pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:22.924292 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:22.924231    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:29:22.924292 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:22.924252    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:29:22.924292 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:22.924265    2582 projected.go:196] Error preparing data for projected volume kube-api-access-s5k9h for pod openshift-network-diagnostics/network-check-target-bkp9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:22.924451 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:22.924326    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h podName:28c8c935-0114-47da-bcf9-4dc1c5642f3d nodeName:}" failed. No retries permitted until 2026-07-13 21:29:24.924302173 +0000 UTC m=+6.080696004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5k9h" (UniqueName: "kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h") pod "network-check-target-bkp9m" (UID: "28c8c935-0114-47da-bcf9-4dc1c5642f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:23.238338 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:23.238237    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 21:24:21 +0000 UTC" deadline="2027-12-10 06:15:28.055833935 +0000 UTC"
Jul 13 21:29:23.238338 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:23.238300    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12344h46m4.817538096s"
Jul 13 21:29:23.358249 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:23.358139    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:23.358249 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:23.358171    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:23.358856 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:23.358337    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:23.358856 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:23.358465    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:23.358856 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:23.358526    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:23.358856 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:23.358611    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:23.387010 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:23.386979    2582 generic.go:356] "Generic (PLEG): container finished" podID="9034b8f06e010d2f76d7e8c886303374" containerID="ade2a294859ddb5f0d4593fe2c974ae2de3a1f1b2f5a3905ec8b53f6db141977" exitCode=0
Jul 13 21:29:23.387145 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:23.387061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal" event={"ID":"9034b8f06e010d2f76d7e8c886303374","Type":"ContainerDied","Data":"ade2a294859ddb5f0d4593fe2c974ae2de3a1f1b2f5a3905ec8b53f6db141977"}
Jul 13 21:29:23.399707 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:23.399335    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal" event={"ID":"f7766334a1c1b5307e0876e60a26fe5a","Type":"ContainerStarted","Data":"fc68fb1803a30b0fc4924cb9b5118da02db9a936690009f93d8adc0f136d4ff7"}
Jul 13 21:29:23.417503 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:23.416741    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-193.ec2.internal" podStartSLOduration=3.416723866 podStartE2EDuration="3.416723866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:23.416426867 +0000 UTC m=+4.572820708" watchObservedRunningTime="2026-07-13 21:29:23.416723866 +0000 UTC m=+4.573117708"
Jul 13 21:29:24.412829 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:24.412795    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal" event={"ID":"9034b8f06e010d2f76d7e8c886303374","Type":"ContainerStarted","Data":"f1feeac0af256d4d467bce3f9a87c1e03c2d5035cf0cfe6087e89d0d13021444"}
Jul 13 21:29:24.838544 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:24.838509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:24.838714 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:24.838583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:24.838714 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:24.838694    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:24.838834 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:24.838753    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret podName:4271181f-f9a6-4bc2-ab11-9e21b0afebd2 nodeName:}" failed. No retries permitted until 2026-07-13 21:29:28.838732637 +0000 UTC m=+9.995126475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret") pod "global-pull-secret-syncer-x6tw2" (UID: "4271181f-f9a6-4bc2-ab11-9e21b0afebd2") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:24.839070 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:24.839055    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:24.839130 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:24.839093    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs podName:71c50b57-6bb8-4e0f-b2a9-1daab13a74cf nodeName:}" failed. No retries permitted until 2026-07-13 21:29:28.839083367 +0000 UTC m=+9.995477185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs") pod "network-metrics-daemon-2njdj" (UID: "71c50b57-6bb8-4e0f-b2a9-1daab13a74cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:24.939600 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:24.939560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5k9h\" (UniqueName: \"kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h\") pod \"network-check-target-bkp9m\" (UID: \"28c8c935-0114-47da-bcf9-4dc1c5642f3d\") " pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:24.939790 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:24.939774    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:29:24.939851 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:24.939803    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:29:24.939851 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:24.939816    2582 projected.go:196] Error preparing data for projected volume kube-api-access-s5k9h for pod openshift-network-diagnostics/network-check-target-bkp9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:24.939956 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:24.939872    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h podName:28c8c935-0114-47da-bcf9-4dc1c5642f3d nodeName:}" failed. No retries permitted until 2026-07-13 21:29:28.93985386 +0000 UTC m=+10.096247678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5k9h" (UniqueName: "kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h") pod "network-check-target-bkp9m" (UID: "28c8c935-0114-47da-bcf9-4dc1c5642f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:25.362836 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:25.362807    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:25.363009 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:25.362991    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:25.363231 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:25.363215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:25.363581 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:25.363386    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:25.363581 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:25.363468    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:25.363581 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:25.363536    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:27.361608 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:27.361570    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:27.362042 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:27.361618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:27.362042 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:27.361579    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:27.362042 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:27.361717    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:27.362042 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:27.361786    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:27.362042 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:27.361882    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:28.873799 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:28.873761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:28.874227 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:28.873836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:28.874227 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:28.873948    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:28.874227 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:28.874001    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs podName:71c50b57-6bb8-4e0f-b2a9-1daab13a74cf nodeName:}" failed. No retries permitted until 2026-07-13 21:29:36.873983475 +0000 UTC m=+18.030377294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs") pod "network-metrics-daemon-2njdj" (UID: "71c50b57-6bb8-4e0f-b2a9-1daab13a74cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:28.874376 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:28.874333    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:28.874416 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:28.874383    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret podName:4271181f-f9a6-4bc2-ab11-9e21b0afebd2 nodeName:}" failed. No retries permitted until 2026-07-13 21:29:36.874372294 +0000 UTC m=+18.030766112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret") pod "global-pull-secret-syncer-x6tw2" (UID: "4271181f-f9a6-4bc2-ab11-9e21b0afebd2") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:28.974965 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:28.974922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5k9h\" (UniqueName: \"kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h\") pod \"network-check-target-bkp9m\" (UID: \"28c8c935-0114-47da-bcf9-4dc1c5642f3d\") " pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:28.975124 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:28.975106    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:29:28.975201 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:28.975132    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:29:28.975201 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:28.975143    2582 projected.go:196] Error preparing data for projected volume kube-api-access-s5k9h for pod openshift-network-diagnostics/network-check-target-bkp9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:28.975267 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:28.975206    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h podName:28c8c935-0114-47da-bcf9-4dc1c5642f3d nodeName:}" failed. No retries permitted until 2026-07-13 21:29:36.975187682 +0000 UTC m=+18.131581518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5k9h" (UniqueName: "kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h") pod "network-check-target-bkp9m" (UID: "28c8c935-0114-47da-bcf9-4dc1c5642f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:29.358780 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:29.358746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:29.358967 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:29.358855    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:29.358967 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:29.358898    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:29.358967 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:29.358940    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:29.359127 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:29.358980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:29.359127 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:29.359045    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:31.357847 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:31.357809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:31.358371 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:31.357949    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:31.358471 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:31.358453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:31.358758 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:31.358593    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:31.358758 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:31.358651    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:31.358758 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:31.358721    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:33.358537 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:33.358504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:33.358537 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:33.358536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:33.358986 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:33.358504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:33.358986 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:33.358651    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:33.358986 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:33.358773    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:33.358986 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:33.358861    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:35.358076 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:35.358041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:35.358594 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:35.358041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:35.358594 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:35.358193    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:35.358594 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:35.358041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:35.358594 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:35.358277    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:35.358594 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:35.358335    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:36.933845 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:36.933814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:36.934289 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:36.933868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:36.934289 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:36.933979    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:36.934289 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:36.934028    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret podName:4271181f-f9a6-4bc2-ab11-9e21b0afebd2 nodeName:}" failed. No retries permitted until 2026-07-13 21:29:52.934013207 +0000 UTC m=+34.090407026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret") pod "global-pull-secret-syncer-x6tw2" (UID: "4271181f-f9a6-4bc2-ab11-9e21b0afebd2") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:36.934289 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:36.933979    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:36.934289 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:36.934077    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs podName:71c50b57-6bb8-4e0f-b2a9-1daab13a74cf nodeName:}" failed. No retries permitted until 2026-07-13 21:29:52.934064209 +0000 UTC m=+34.090458030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs") pod "network-metrics-daemon-2njdj" (UID: "71c50b57-6bb8-4e0f-b2a9-1daab13a74cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:37.035132 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:37.035094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5k9h\" (UniqueName: \"kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h\") pod \"network-check-target-bkp9m\" (UID: \"28c8c935-0114-47da-bcf9-4dc1c5642f3d\") " pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:37.035302 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:37.035272    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:29:37.035302 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:37.035294    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:29:37.035395 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:37.035307    2582 projected.go:196] Error preparing data for projected volume kube-api-access-s5k9h for pod openshift-network-diagnostics/network-check-target-bkp9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:37.035395 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:37.035373    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h podName:28c8c935-0114-47da-bcf9-4dc1c5642f3d nodeName:}" failed. No retries permitted until 2026-07-13 21:29:53.035353708 +0000 UTC m=+34.191747529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5k9h" (UniqueName: "kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h") pod "network-check-target-bkp9m" (UID: "28c8c935-0114-47da-bcf9-4dc1c5642f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:37.358405 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:37.358208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:37.358405 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:37.358244    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:37.358405 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:37.358208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:37.358405 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:37.358331    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:37.358405 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:37.358412    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:37.358764 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:37.358521    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:39.358937 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:39.358889    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:39.359420 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:39.359017    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:39.359420 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:39.359025    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:39.359420 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:39.359166    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:39.359420 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:39.359236    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:39.359420 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:39.359335    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:41.358041 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.358016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:41.358678 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.358044    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:41.358678 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.358018    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:41.358678 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:41.358138    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:41.358678 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:41.358259    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:41.358678 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:41.358371    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:41.442402 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.442272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tbqlb" event={"ID":"721b2fc5-6fe6-49a2-abaf-ea836e1d93bb","Type":"ContainerStarted","Data":"11132e354fff580fa1a2f985cec6138ba831580147ebbbee5ff924f635d2f1d9"}
Jul 13 21:29:41.443608 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.443588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rh74n" event={"ID":"4473539f-ed4b-437c-9814-8ee29ac74bf6","Type":"ContainerStarted","Data":"4a3de371901f083b1b64e7b33c3ef388a26de1ed0c988bb3521fdf37541cc245"}
Jul 13 21:29:41.444929 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.444910    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq" event={"ID":"f59c65a1-d32f-4b56-9217-c5bab1c467a9","Type":"ContainerStarted","Data":"f9596b7b9e7bc0f6b0ca5bfc03dba34ac2f9dccfc295ebc9e571e6895e6541e9"}
Jul 13 21:29:41.446371 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.446339    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7b5bb8c-8520-4b35-8657-b66b7e5be404" containerID="8a54f3606095f321a4602f477fc902b20e78ce56e10984634501f8482a805a4c" exitCode=0
Jul 13 21:29:41.446444 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.446411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf9hl" event={"ID":"a7b5bb8c-8520-4b35-8657-b66b7e5be404","Type":"ContainerDied","Data":"8a54f3606095f321a4602f477fc902b20e78ce56e10984634501f8482a805a4c"}
Jul 13 21:29:41.447722 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.447705    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8hglh" event={"ID":"1da0ad36-59f6-439c-b099-dc6884ff2431","Type":"ContainerStarted","Data":"d76933ddeb8500891a5e0dec636d6f0a23e533d74db63131da257bf970b16ba9"}
Jul 13 21:29:41.448961 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.448937    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pmd7k" event={"ID":"9dd52d08-2a36-4291-ad30-52d7bb0ba254","Type":"ContainerStarted","Data":"9fe6de5f22a3b7adf45d8bd8e0d2bf0fe68470e1a95a782c291fa58b591de1e2"}
Jul 13 21:29:41.450436 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.450413    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-778w5" event={"ID":"8fce6eb3-9db6-42b2-b5e1-0a7d1a1c4e4d","Type":"ContainerStarted","Data":"02cdb7ac5db3af64ecfa30c397abec92cfffaedadfc51a441493b33a30d2ea16"}
Jul 13 21:29:41.452908 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.452881    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:29:41.453254 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.453234    2582 generic.go:356] "Generic (PLEG): container finished" podID="0fb821cc-e52a-468e-b50b-afc5e461d6c5" containerID="61bec0a1e44f4f727ec85941b52d915b151fbed9811d0203b91e219c3a3a8aeb" exitCode=1
Jul 13 21:29:41.453340 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.453262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" event={"ID":"0fb821cc-e52a-468e-b50b-afc5e461d6c5","Type":"ContainerStarted","Data":"0f0993ae9c5c0e244ede0f5dcc9154435ff34a05faa6e61031441e98a2a7157b"}
Jul 13 21:29:41.453340 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.453312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" event={"ID":"0fb821cc-e52a-468e-b50b-afc5e461d6c5","Type":"ContainerStarted","Data":"923b1d61f70285492fa4b9f9bcdab8ae84cc35a1a79d017360fed3e8430963ed"}
Jul 13 21:29:41.453461 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.453341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" event={"ID":"0fb821cc-e52a-468e-b50b-afc5e461d6c5","Type":"ContainerStarted","Data":"934b67d33f26044809c1ca75e99cdc11da3a88d60128d48f5d257bb7d4a83d07"}
Jul 13 21:29:41.453461 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.453369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" event={"ID":"0fb821cc-e52a-468e-b50b-afc5e461d6c5","Type":"ContainerDied","Data":"61bec0a1e44f4f727ec85941b52d915b151fbed9811d0203b91e219c3a3a8aeb"}
Jul 13 21:29:41.453461 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.453396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" event={"ID":"0fb821cc-e52a-468e-b50b-afc5e461d6c5","Type":"ContainerStarted","Data":"a6b6d2199f728db5f0f83480f8e726372d4965d08be09530ca6c379021ccf4f3"}
Jul 13 21:29:41.456558 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.456503    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tbqlb" podStartSLOduration=3.8909589 podStartE2EDuration="22.456487285s" totalImagesPullingTime="18.565528385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:41.456091338 +0000 UTC m=+22.612485180" watchObservedRunningTime="2026-07-13 21:29:41.456487285 +0000 UTC m=+22.612881126"
Jul 13 21:29:41.456655 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.456630    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-193.ec2.internal" podStartSLOduration=21.456615816 podStartE2EDuration="21.456615816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:24.427574494 +0000 UTC m=+5.583968346" watchObservedRunningTime="2026-07-13 21:29:41.456615816 +0000 UTC m=+22.613009661"
Jul 13 21:29:41.473135 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.473086    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-778w5" podStartSLOduration=3.764025142 podStartE2EDuration="22.473071356s" totalImagesPullingTime="18.709046214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:41.472309878 +0000 UTC m=+22.628703729" watchObservedRunningTime="2026-07-13 21:29:41.473071356 +0000 UTC m=+22.629465196"
Jul 13 21:29:41.487633 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.487596    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rh74n" podStartSLOduration=3.920344854 podStartE2EDuration="22.487588188s" totalImagesPullingTime="18.567243334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:41.487372953 +0000 UTC m=+22.643766793" watchObservedRunningTime="2026-07-13 21:29:41.487588188 +0000 UTC m=+22.643982028"
Jul 13 21:29:41.521855 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.521820    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pmd7k" podStartSLOduration=4.062585737 podStartE2EDuration="22.521807778s" totalImagesPullingTime="18.459222041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:41.52178828 +0000 UTC m=+22.678182131" watchObservedRunningTime="2026-07-13 21:29:41.521807778 +0000 UTC m=+22.678201680"
Jul 13 21:29:41.539857 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:41.539817    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8hglh" podStartSLOduration=3.976544129 podStartE2EDuration="22.539799808s" totalImagesPullingTime="18.563255679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:41.539185455 +0000 UTC m=+22.695579295" watchObservedRunningTime="2026-07-13 21:29:41.539799808 +0000 UTC m=+22.696193648"
Jul 13 21:29:42.313292 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:42.313266    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 21:29:42.458414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:42.458323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq" event={"ID":"f59c65a1-d32f-4b56-9217-c5bab1c467a9","Type":"ContainerStarted","Data":"e233eec7cd423dd42f121856d7587bec6a1f908d21c3bdd14d798e90e0ae457e"}
Jul 13 21:29:42.460246 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:42.460211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kdnjq" event={"ID":"dcedcbc0-ec41-414b-a6fa-3fa67b1f2a81","Type":"ContainerStarted","Data":"f45a41d91368f61e1aebc578efcb32acd90dbb7b575b410ba92d08956ef37d5e"}
Jul 13 21:29:42.463430 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:42.463407    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:29:42.463834 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:42.463736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" event={"ID":"0fb821cc-e52a-468e-b50b-afc5e461d6c5","Type":"ContainerStarted","Data":"3416e67e1030dad90ff1c576dcfd7ff37227d59fcb6ce24672ed5c8d9f932252"}
Jul 13 21:29:42.476622 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:42.476581    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kdnjq" podStartSLOduration=4.901999167 podStartE2EDuration="23.476568695s" totalImagesPullingTime="18.574569528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:42.47570802 +0000 UTC m=+23.632101860" watchObservedRunningTime="2026-07-13 21:29:42.476568695 +0000 UTC m=+23.632962535"
Jul 13 21:29:43.285728 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:43.285645    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T21:29:42.313288887Z","UUID":"822ec942-f2bf-4552-88c6-fed07aaa3460","Handler":null,"Name":"","Endpoint":""}
Jul 13 21:29:43.287053 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:43.287033    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 21:29:43.287181 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:43.287067    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 21:29:43.357983 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:43.357952    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:43.358249 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:43.357958    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:43.358249 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:43.358067    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:43.358249 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:43.357958    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:43.358249 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:43.358142    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:43.358249 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:43.358221    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:44.167787 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:44.167601    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:44.168351 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:44.168108    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:44.471551 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:44.471472    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq" event={"ID":"f59c65a1-d32f-4b56-9217-c5bab1c467a9","Type":"ContainerStarted","Data":"e5270761d213d97b1fd8c93f23a30bff20eb1ab99f5745aea8fe4ff2f3e90644"}
Jul 13 21:29:44.474887 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:44.474869    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:29:44.475286 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:44.475252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" event={"ID":"0fb821cc-e52a-468e-b50b-afc5e461d6c5","Type":"ContainerStarted","Data":"610a42317e49c37190503dd4eac87c5a2b66cdf062b55c868259053515b6b078"}
Jul 13 21:29:44.475493 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:44.475473    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:44.475919 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:44.475900    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pmd7k"
Jul 13 21:29:44.489343 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:44.489297    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rq6kq" podStartSLOduration=4.7497824170000005 podStartE2EDuration="25.48927924s" totalImagesPullingTime="20.739496823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:44.488817576 +0000 UTC m=+25.645211417" watchObservedRunningTime="2026-07-13 21:29:44.48927924 +0000 UTC m=+25.645673086"
Jul 13 21:29:45.358245 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:45.358214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:45.358837 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:45.358214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:45.358837 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:45.358346    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:45.358837 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:45.358436    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:45.358837 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:45.358214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:45.358837 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:45.358529    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:46.484119 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:46.483937    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7b5bb8c-8520-4b35-8657-b66b7e5be404" containerID="d1f4b76a8c03a22add6ed653f16e81c8062e7a76c8302fd321d82e32e0376274" exitCode=0
Jul 13 21:29:46.484700 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:46.484012    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf9hl" event={"ID":"a7b5bb8c-8520-4b35-8657-b66b7e5be404","Type":"ContainerDied","Data":"d1f4b76a8c03a22add6ed653f16e81c8062e7a76c8302fd321d82e32e0376274"}
Jul 13 21:29:46.487760 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:46.487741    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:29:46.488103 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:46.488083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" event={"ID":"0fb821cc-e52a-468e-b50b-afc5e461d6c5","Type":"ContainerStarted","Data":"9b4ee89763be5d661dc28768c5158c6514d6052467ded371f6386c7927884ee4"}
Jul 13 21:29:46.488528 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:46.488511    2582 scope.go:117] "RemoveContainer" containerID="61bec0a1e44f4f727ec85941b52d915b151fbed9811d0203b91e219c3a3a8aeb"
Jul 13 21:29:47.358094 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.358064    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:47.358094 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.358075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:47.358323 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:47.358182    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:47.358323 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:47.358239    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:47.358323 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.358272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:47.358449 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:47.358365    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:47.495644 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.495618    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:29:47.496066 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.496039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" event={"ID":"0fb821cc-e52a-468e-b50b-afc5e461d6c5","Type":"ContainerStarted","Data":"3041a90cec915d84bba2f0da67dcd2298ef245dce809b740d6fa88dd73f2feec"}
Jul 13 21:29:47.496390 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.496370    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:47.496517 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.496456    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:47.496575 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.496558    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:47.513416 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.513390    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:47.514449 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.514403    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:29:47.524348 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:47.524304    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k" podStartSLOduration=9.915388443 podStartE2EDuration="28.524287555s" totalImagesPullingTime="18.608899112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:47.523718528 +0000 UTC m=+28.680112379" watchObservedRunningTime="2026-07-13 21:29:47.524287555 +0000 UTC m=+28.680681395"
Jul 13 21:29:48.062901 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:48.062680    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x6tw2"]
Jul 13 21:29:48.063037 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:48.062948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:48.063074 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:48.063036    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:48.065822 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:48.065794    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bkp9m"]
Jul 13 21:29:48.065957 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:48.065902    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:48.066031 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:48.066009    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:48.066433 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:48.066413    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2njdj"]
Jul 13 21:29:48.066515 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:48.066504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:48.066618 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:48.066598    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:48.499917 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:48.499886    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7b5bb8c-8520-4b35-8657-b66b7e5be404" containerID="7f7f0048a5c209263fc9e6c66ac590342acf90b835fdc3b7bab67a026c1f59a1" exitCode=0
Jul 13 21:29:48.500398 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:48.499959    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf9hl" event={"ID":"a7b5bb8c-8520-4b35-8657-b66b7e5be404","Type":"ContainerDied","Data":"7f7f0048a5c209263fc9e6c66ac590342acf90b835fdc3b7bab67a026c1f59a1"}
Jul 13 21:29:49.358739 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:49.358714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:49.358900 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:49.358825    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:50.358584 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:50.358505    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:50.358584 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:50.358539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:50.359405 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:50.358604    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:50.359405 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:50.358666    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:50.506948 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:50.506915    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7b5bb8c-8520-4b35-8657-b66b7e5be404" containerID="2e914966b456e1791388e7ec72a93395eb6efda647acc4fdaafee1c3de9f38ad" exitCode=0
Jul 13 21:29:50.507169 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:50.506979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf9hl" event={"ID":"a7b5bb8c-8520-4b35-8657-b66b7e5be404","Type":"ContainerDied","Data":"2e914966b456e1791388e7ec72a93395eb6efda647acc4fdaafee1c3de9f38ad"}
Jul 13 21:29:51.357974 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:51.357942    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:51.358173 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:51.358054    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:52.358374 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:52.358346    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:52.358835 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:52.358355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:52.358835 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:52.358454    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6tw2" podUID="4271181f-f9a6-4bc2-ab11-9e21b0afebd2"
Jul 13 21:29:52.358835 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:52.358558    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:29:52.964357 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:52.964320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:52.964520 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:52.964376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:52.964520 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:52.964485    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:52.964520 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:52.964487    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:52.964671 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:52.964538    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret podName:4271181f-f9a6-4bc2-ab11-9e21b0afebd2 nodeName:}" failed. No retries permitted until 2026-07-13 21:30:24.964520646 +0000 UTC m=+66.120914484 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret") pod "global-pull-secret-syncer-x6tw2" (UID: "4271181f-f9a6-4bc2-ab11-9e21b0afebd2") : object "kube-system"/"original-pull-secret" not registered
Jul 13 21:29:52.964671 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:52.964552    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs podName:71c50b57-6bb8-4e0f-b2a9-1daab13a74cf nodeName:}" failed. No retries permitted until 2026-07-13 21:30:24.964545742 +0000 UTC m=+66.120939560 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs") pod "network-metrics-daemon-2njdj" (UID: "71c50b57-6bb8-4e0f-b2a9-1daab13a74cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 21:29:53.064797 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.064765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5k9h\" (UniqueName: \"kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h\") pod \"network-check-target-bkp9m\" (UID: \"28c8c935-0114-47da-bcf9-4dc1c5642f3d\") " pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:53.064947 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:53.064909    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 21:29:53.064947 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:53.064930    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 21:29:53.064947 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:53.064940    2582 projected.go:196] Error preparing data for projected volume kube-api-access-s5k9h for pod openshift-network-diagnostics/network-check-target-bkp9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:53.065052 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:53.064985    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h podName:28c8c935-0114-47da-bcf9-4dc1c5642f3d nodeName:}" failed. No retries permitted until 2026-07-13 21:30:25.064972246 +0000 UTC m=+66.221366065 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5k9h" (UniqueName: "kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h") pod "network-check-target-bkp9m" (UID: "28c8c935-0114-47da-bcf9-4dc1c5642f3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 21:29:53.357785 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.357756    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:53.357969 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:53.357875    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bkp9m" podUID="28c8c935-0114-47da-bcf9-4dc1c5642f3d"
Jul 13 21:29:53.606600 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.606383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:29:53.687406 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.687294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:29:53.713808 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.713782    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-193.ec2.internal" event="NodeReady"
Jul 13 21:29:53.713942 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.713901    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 21:29:53.755860 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.755832    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tvf8f"]
Jul 13 21:29:53.777585 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.777558    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9znbk"]
Jul 13 21:29:53.777713 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.777688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.780190 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.780169    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 21:29:53.780190 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.780180    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 21:29:53.780349 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.780195    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l6v89\""
Jul 13 21:29:53.787576 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.787557    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tvf8f"]
Jul 13 21:29:53.787678 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.787587    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9znbk"]
Jul 13 21:29:53.787678 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.787664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9znbk"
Jul 13 21:29:53.789868 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.789849    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 21:29:53.789963 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.789933    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 21:29:53.790037 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.790019    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wlljv\""
Jul 13 21:29:53.790178 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.790129    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 21:29:53.870870 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.870843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.870990 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.870903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwtqv\" (UniqueName: \"kubernetes.io/projected/7e46539f-e0bc-4eea-a0ca-e65633ccf553-kube-api-access-bwtqv\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.870990 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.870964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e46539f-e0bc-4eea-a0ca-e65633ccf553-tmp-dir\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.871112 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.871010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e46539f-e0bc-4eea-a0ca-e65633ccf553-config-volume\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.876627 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.876602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:29:53.971862 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.971747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.971862 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.971854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3d9f579-5163-4bf3-906e-e497ca6d4d89-cert\") pod \"ingress-canary-9znbk\" (UID: \"f3d9f579-5163-4bf3-906e-e497ca6d4d89\") " pod="openshift-ingress-canary/ingress-canary-9znbk"
Jul 13 21:29:53.972031 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.971892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwtqv\" (UniqueName: \"kubernetes.io/projected/7e46539f-e0bc-4eea-a0ca-e65633ccf553-kube-api-access-bwtqv\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.972031 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.971914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e46539f-e0bc-4eea-a0ca-e65633ccf553-tmp-dir\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.972031 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.971935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mklk7\" (UniqueName: \"kubernetes.io/projected/f3d9f579-5163-4bf3-906e-e497ca6d4d89-kube-api-access-mklk7\") pod \"ingress-canary-9znbk\" (UID: \"f3d9f579-5163-4bf3-906e-e497ca6d4d89\") " pod="openshift-ingress-canary/ingress-canary-9znbk"
Jul 13 21:29:53.972031 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.971960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e46539f-e0bc-4eea-a0ca-e65633ccf553-config-volume\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.972224 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:53.971913    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:29:53.972285 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:53.972243    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls podName:7e46539f-e0bc-4eea-a0ca-e65633ccf553 nodeName:}" failed. No retries permitted until 2026-07-13 21:29:54.472225961 +0000 UTC m=+35.628619779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls") pod "dns-default-tvf8f" (UID: "7e46539f-e0bc-4eea-a0ca-e65633ccf553") : secret "dns-default-metrics-tls" not found
Jul 13 21:29:53.972466 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.972442    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e46539f-e0bc-4eea-a0ca-e65633ccf553-tmp-dir\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.972583 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.972558    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e46539f-e0bc-4eea-a0ca-e65633ccf553-config-volume\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:53.979277 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.979251    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:29:53.983891 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:53.983863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwtqv\" (UniqueName: \"kubernetes.io/projected/7e46539f-e0bc-4eea-a0ca-e65633ccf553-kube-api-access-bwtqv\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:54.072752 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.072722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3d9f579-5163-4bf3-906e-e497ca6d4d89-cert\") pod \"ingress-canary-9znbk\" (UID: \"f3d9f579-5163-4bf3-906e-e497ca6d4d89\") " pod="openshift-ingress-canary/ingress-canary-9znbk"
Jul 13 21:29:54.072894 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.072767    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mklk7\" (UniqueName: \"kubernetes.io/projected/f3d9f579-5163-4bf3-906e-e497ca6d4d89-kube-api-access-mklk7\") pod \"ingress-canary-9znbk\" (UID: \"f3d9f579-5163-4bf3-906e-e497ca6d4d89\") " pod="openshift-ingress-canary/ingress-canary-9znbk"
Jul 13 21:29:54.075279 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.075259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3d9f579-5163-4bf3-906e-e497ca6d4d89-cert\") pod \"ingress-canary-9znbk\" (UID: \"f3d9f579-5163-4bf3-906e-e497ca6d4d89\") " pod="openshift-ingress-canary/ingress-canary-9znbk"
Jul 13 21:29:54.081445 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.081425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mklk7\" (UniqueName: \"kubernetes.io/projected/f3d9f579-5163-4bf3-906e-e497ca6d4d89-kube-api-access-mklk7\") pod \"ingress-canary-9znbk\" (UID: \"f3d9f579-5163-4bf3-906e-e497ca6d4d89\") " pod="openshift-ingress-canary/ingress-canary-9znbk"
Jul 13 21:29:54.098198 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.098176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9znbk"
Jul 13 21:29:54.358221 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.358193    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:29:54.358409 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.358198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:29:54.360762 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.360741    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89gzj\""
Jul 13 21:29:54.360867 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.360784    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 21:29:54.360867 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.360740    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 21:29:54.476311 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.476276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:54.476554 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:54.476534    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:29:54.476627 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:54.476617    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls podName:7e46539f-e0bc-4eea-a0ca-e65633ccf553 nodeName:}" failed. No retries permitted until 2026-07-13 21:29:55.47660072 +0000 UTC m=+36.632994541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls") pod "dns-default-tvf8f" (UID: "7e46539f-e0bc-4eea-a0ca-e65633ccf553") : secret "dns-default-metrics-tls" not found
Jul 13 21:29:54.852053 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:54.852022    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9znbk"]
Jul 13 21:29:54.976883 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:29:54.976829    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d9f579_5163_4bf3_906e_e497ca6d4d89.slice/crio-6349c3dc5e2dffc97badcc65fedc86c80fe913e302023bf25121b727deada0c2 WatchSource:0}: Error finding container 6349c3dc5e2dffc97badcc65fedc86c80fe913e302023bf25121b727deada0c2: Status 404 returned error can't find the container with id 6349c3dc5e2dffc97badcc65fedc86c80fe913e302023bf25121b727deada0c2
Jul 13 21:29:55.358464 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:55.358437    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:29:55.361109 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:55.361026    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 21:29:55.361256 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:55.361126    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 21:29:55.361256 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:55.361181    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qsrtw\""
Jul 13 21:29:55.483166 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:55.482959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:55.483166 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:55.483096    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:29:55.483298 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:55.483214    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls podName:7e46539f-e0bc-4eea-a0ca-e65633ccf553 nodeName:}" failed. No retries permitted until 2026-07-13 21:29:57.483200247 +0000 UTC m=+38.639594066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls") pod "dns-default-tvf8f" (UID: "7e46539f-e0bc-4eea-a0ca-e65633ccf553") : secret "dns-default-metrics-tls" not found
Jul 13 21:29:55.520608 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:55.520579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9znbk" event={"ID":"f3d9f579-5163-4bf3-906e-e497ca6d4d89","Type":"ContainerStarted","Data":"6349c3dc5e2dffc97badcc65fedc86c80fe913e302023bf25121b727deada0c2"}
Jul 13 21:29:55.523134 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:55.523113    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7b5bb8c-8520-4b35-8657-b66b7e5be404" containerID="406954f3db63841a40fe17d5743bf789e5046c1ea310addc0933347e89f0ecbe" exitCode=0
Jul 13 21:29:55.523226 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:55.523144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf9hl" event={"ID":"a7b5bb8c-8520-4b35-8657-b66b7e5be404","Type":"ContainerDied","Data":"406954f3db63841a40fe17d5743bf789e5046c1ea310addc0933347e89f0ecbe"}
Jul 13 21:29:56.528557 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:56.528526    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7b5bb8c-8520-4b35-8657-b66b7e5be404" containerID="480048a23e599c802ada904bacac106e36aa57e5a70cdb65e21a364ecb116030" exitCode=0
Jul 13 21:29:56.528889 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:56.528580    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf9hl" event={"ID":"a7b5bb8c-8520-4b35-8657-b66b7e5be404","Type":"ContainerDied","Data":"480048a23e599c802ada904bacac106e36aa57e5a70cdb65e21a364ecb116030"}
Jul 13 21:29:57.500791 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:57.500758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:29:57.500968 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:57.500911    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:29:57.501033 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:29:57.500972    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls podName:7e46539f-e0bc-4eea-a0ca-e65633ccf553 nodeName:}" failed. No retries permitted until 2026-07-13 21:30:01.500955686 +0000 UTC m=+42.657349524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls") pod "dns-default-tvf8f" (UID: "7e46539f-e0bc-4eea-a0ca-e65633ccf553") : secret "dns-default-metrics-tls" not found
Jul 13 21:29:57.533719 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:57.533685    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf9hl" event={"ID":"a7b5bb8c-8520-4b35-8657-b66b7e5be404","Type":"ContainerStarted","Data":"491cc1661693d7f67812648bca7767bfbb9a72f39d1cd7723f6d9ea021a42e0c"}
Jul 13 21:29:57.535046 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:57.535023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9znbk" event={"ID":"f3d9f579-5163-4bf3-906e-e497ca6d4d89","Type":"ContainerStarted","Data":"f65b906d8680844fd5495c09730dfb41c82bef39178f072cc7b21eb7236dfe06"}
Jul 13 21:29:57.555842 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:57.555807    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wf9hl" podStartSLOduration=8.316473145 podStartE2EDuration="38.555795887s" totalImagesPullingTime="30.239322742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:57.554979839 +0000 UTC m=+38.711373679" watchObservedRunningTime="2026-07-13 21:29:57.555795887 +0000 UTC m=+38.712189727"
Jul 13 21:29:57.570321 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:57.570287    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9znbk" podStartSLOduration=2.331079924 podStartE2EDuration="4.57027562s" totalImagesPullingTime="2.239195696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:29:57.569828846 +0000 UTC m=+38.726222688" watchObservedRunningTime="2026-07-13 21:29:57.57027562 +0000 UTC m=+38.726669454"
Jul 13 21:29:57.648473 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:57.648447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:29:57.708985 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:57.708957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:29:57.878082 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:57.878004    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:29:57.930988 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:57.930954    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:29:58.472631 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:58.472606    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:29:58.999458 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:29:58.999426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:00.573594 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:00.573553    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:00.674915 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:00.674893    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:00.775172 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:00.775135    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:01.525989 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:01.525941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:30:01.526195 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:30:01.526096    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:30:01.526195 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:30:01.526183    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls podName:7e46539f-e0bc-4eea-a0ca-e65633ccf553 nodeName:}" failed. No retries permitted until 2026-07-13 21:30:09.526143296 +0000 UTC m=+50.682537115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls") pod "dns-default-tvf8f" (UID: "7e46539f-e0bc-4eea-a0ca-e65633ccf553") : secret "dns-default-metrics-tls" not found
Jul 13 21:30:01.578352 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:01.578306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:01.678819 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:01.678786    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:01.731658 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:01.731628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:05.547870 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:05.547836    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:05.621993 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:05.621963    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:09.580946 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:09.580912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:30:09.581361 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:30:09.581030    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:30:09.581361 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:30:09.581078    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls podName:7e46539f-e0bc-4eea-a0ca-e65633ccf553 nodeName:}" failed. No retries permitted until 2026-07-13 21:30:25.581063115 +0000 UTC m=+66.737456933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls") pod "dns-default-tvf8f" (UID: "7e46539f-e0bc-4eea-a0ca-e65633ccf553") : secret "dns-default-metrics-tls" not found
Jul 13 21:30:14.087308 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:14.087279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:14.188705 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:14.188675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:18.443880 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:18.443847    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:18.497817 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:18.497788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:18.651435 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:18.651406    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:18.752436 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:18.752409    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:19.512346 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:19.512323    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm77k"
Jul 13 21:30:19.659202 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:19.659172    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:19.723869 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:19.723835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:19.860538 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:19.860456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:19.913394 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:19.913359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:19.997228 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:19.997201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:20.050742 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:20.050713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:24.982270 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:24.982229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:30:24.982656 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:24.982293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:30:24.985025 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:24.984994    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 21:30:24.985025 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:24.985012    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 21:30:24.993348 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:30:24.993326    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 13 21:30:24.993423 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:30:24.993411    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs podName:71c50b57-6bb8-4e0f-b2a9-1daab13a74cf nodeName:}" failed. No retries permitted until 2026-07-13 21:31:28.993390046 +0000 UTC m=+130.149783880 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs") pod "network-metrics-daemon-2njdj" (UID: "71c50b57-6bb8-4e0f-b2a9-1daab13a74cf") : secret "metrics-daemon-secret" not found
Jul 13 21:30:24.996456 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:24.996441    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4271181f-f9a6-4bc2-ab11-9e21b0afebd2-original-pull-secret\") pod \"global-pull-secret-syncer-x6tw2\" (UID: \"4271181f-f9a6-4bc2-ab11-9e21b0afebd2\") " pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:30:25.083407 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.083374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5k9h\" (UniqueName: \"kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h\") pod \"network-check-target-bkp9m\" (UID: \"28c8c935-0114-47da-bcf9-4dc1c5642f3d\") " pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:30:25.085971 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.085955    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 21:30:25.095926 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.095910    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 21:30:25.106993 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.106963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5k9h\" (UniqueName: \"kubernetes.io/projected/28c8c935-0114-47da-bcf9-4dc1c5642f3d-kube-api-access-s5k9h\") pod \"network-check-target-bkp9m\" (UID: \"28c8c935-0114-47da-bcf9-4dc1c5642f3d\") " pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:30:25.269932 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.269846    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6tw2"
Jul 13 21:30:25.369942 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.369919    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qsrtw\""
Jul 13 21:30:25.377947 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.377929    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:30:25.384758 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.384731    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x6tw2"]
Jul 13 21:30:25.388922 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:30:25.388901    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4271181f_f9a6_4bc2_ab11_9e21b0afebd2.slice/crio-7e145f27545b74ff20b3cf99d27a492448fa1af9734b41621f6e43ff4b7e353a WatchSource:0}: Error finding container 7e145f27545b74ff20b3cf99d27a492448fa1af9734b41621f6e43ff4b7e353a: Status 404 returned error can't find the container with id 7e145f27545b74ff20b3cf99d27a492448fa1af9734b41621f6e43ff4b7e353a
Jul 13 21:30:25.502534 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.502511    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bkp9m"]
Jul 13 21:30:25.504481 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:30:25.504455    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c8c935_0114_47da_bcf9_4dc1c5642f3d.slice/crio-50f7091c972f8e642f9b4a8fda906b450b0dc2cdb4e3165ceb8b89006a6923fd WatchSource:0}: Error finding container 50f7091c972f8e642f9b4a8fda906b450b0dc2cdb4e3165ceb8b89006a6923fd: Status 404 returned error can't find the container with id 50f7091c972f8e642f9b4a8fda906b450b0dc2cdb4e3165ceb8b89006a6923fd
Jul 13 21:30:25.587975 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.587919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:30:25.588069 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:30:25.588054    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:30:25.588115 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:30:25.588105    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls podName:7e46539f-e0bc-4eea-a0ca-e65633ccf553 nodeName:}" failed. No retries permitted until 2026-07-13 21:30:57.588091615 +0000 UTC m=+98.744485437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls") pod "dns-default-tvf8f" (UID: "7e46539f-e0bc-4eea-a0ca-e65633ccf553") : secret "dns-default-metrics-tls" not found
Jul 13 21:30:25.596958 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.596931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bkp9m" event={"ID":"28c8c935-0114-47da-bcf9-4dc1c5642f3d","Type":"ContainerStarted","Data":"50f7091c972f8e642f9b4a8fda906b450b0dc2cdb4e3165ceb8b89006a6923fd"}
Jul 13 21:30:25.597981 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:25.597962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x6tw2" event={"ID":"4271181f-f9a6-4bc2-ab11-9e21b0afebd2","Type":"ContainerStarted","Data":"7e145f27545b74ff20b3cf99d27a492448fa1af9734b41621f6e43ff4b7e353a"}
Jul 13 21:30:30.614491 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:30.614384    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x6tw2" event={"ID":"4271181f-f9a6-4bc2-ab11-9e21b0afebd2","Type":"ContainerStarted","Data":"d8293624273a27103fb588ad8332cec2535be580e75dd5bc8ab13d8b102436f0"}
Jul 13 21:30:30.618503 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:30.618477    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bkp9m" event={"ID":"28c8c935-0114-47da-bcf9-4dc1c5642f3d","Type":"ContainerStarted","Data":"b8248ed522b3dc231cf20eeeaec0d5e5471a94fc2a44bc8cffee22f4f7461177"}
Jul 13 21:30:30.618639 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:30.618606    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:30:30.628694 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:30.628645    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x6tw2" podStartSLOduration=66.648786217 podStartE2EDuration="1m11.628630753s" totalImagesPullingTime="4.979844536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:30:30.628062391 +0000 UTC m=+71.784456234" watchObservedRunningTime="2026-07-13 21:30:30.628630753 +0000 UTC m=+71.785024592"
Jul 13 21:30:30.642554 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:30.642511    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bkp9m" podStartSLOduration=66.788847058 podStartE2EDuration="1m11.642497229s" totalImagesPullingTime="4.853650171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:30:30.641422795 +0000 UTC m=+71.797816635" watchObservedRunningTime="2026-07-13 21:30:30.642497229 +0000 UTC m=+71.798891068"
Jul 13 21:30:40.457281 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.457242    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"]
Jul 13 21:30:40.460433 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.460413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.462741 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.462720    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 13 21:30:40.462999 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.462980    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 13 21:30:40.463070 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.463056    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 13 21:30:40.464430 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.464412    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 13 21:30:40.470035 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.470013    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"]
Jul 13 21:30:40.587592 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.587564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/626d24de-d296-4890-bf60-10c033ae84be-tmp\") pod \"klusterlet-addon-workmgr-5959874999-s9gv9\" (UID: \"626d24de-d296-4890-bf60-10c033ae84be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.587717 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.587626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwjsp\" (UniqueName: \"kubernetes.io/projected/626d24de-d296-4890-bf60-10c033ae84be-kube-api-access-jwjsp\") pod \"klusterlet-addon-workmgr-5959874999-s9gv9\" (UID: \"626d24de-d296-4890-bf60-10c033ae84be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.587717 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.587650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/626d24de-d296-4890-bf60-10c033ae84be-klusterlet-config\") pod \"klusterlet-addon-workmgr-5959874999-s9gv9\" (UID: \"626d24de-d296-4890-bf60-10c033ae84be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.612186 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.612146    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:40.664676 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.664654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:30:40.688471 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.688441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/626d24de-d296-4890-bf60-10c033ae84be-tmp\") pod \"klusterlet-addon-workmgr-5959874999-s9gv9\" (UID: \"626d24de-d296-4890-bf60-10c033ae84be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.688566 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.688500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwjsp\" (UniqueName: \"kubernetes.io/projected/626d24de-d296-4890-bf60-10c033ae84be-kube-api-access-jwjsp\") pod \"klusterlet-addon-workmgr-5959874999-s9gv9\" (UID: \"626d24de-d296-4890-bf60-10c033ae84be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.688689 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.688668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/626d24de-d296-4890-bf60-10c033ae84be-klusterlet-config\") pod \"klusterlet-addon-workmgr-5959874999-s9gv9\" (UID: \"626d24de-d296-4890-bf60-10c033ae84be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.688863 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.688845    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/626d24de-d296-4890-bf60-10c033ae84be-tmp\") pod \"klusterlet-addon-workmgr-5959874999-s9gv9\" (UID: \"626d24de-d296-4890-bf60-10c033ae84be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.691058 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.691038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/626d24de-d296-4890-bf60-10c033ae84be-klusterlet-config\") pod \"klusterlet-addon-workmgr-5959874999-s9gv9\" (UID: \"626d24de-d296-4890-bf60-10c033ae84be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.701726 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.701706    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwjsp\" (UniqueName: \"kubernetes.io/projected/626d24de-d296-4890-bf60-10c033ae84be-kube-api-access-jwjsp\") pod \"klusterlet-addon-workmgr-5959874999-s9gv9\" (UID: \"626d24de-d296-4890-bf60-10c033ae84be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.768723 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.768703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:40.882831 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:40.882802    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"]
Jul 13 21:30:40.886740 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:30:40.886712    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626d24de_d296_4890_bf60_10c033ae84be.slice/crio-dcca9640e26c23cf99c67f9ab5ffdc8f2fe89ea2c944695ed997faa69949ba00 WatchSource:0}: Error finding container dcca9640e26c23cf99c67f9ab5ffdc8f2fe89ea2c944695ed997faa69949ba00: Status 404 returned error can't find the container with id dcca9640e26c23cf99c67f9ab5ffdc8f2fe89ea2c944695ed997faa69949ba00
Jul 13 21:30:41.647768 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:41.647620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9" event={"ID":"626d24de-d296-4890-bf60-10c033ae84be","Type":"ContainerStarted","Data":"dcca9640e26c23cf99c67f9ab5ffdc8f2fe89ea2c944695ed997faa69949ba00"}
Jul 13 21:30:44.656663 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:44.656583    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9" event={"ID":"626d24de-d296-4890-bf60-10c033ae84be","Type":"ContainerStarted","Data":"e754014228aef97264c3815dea9293b910477ae3d076770780cf16475cd0b78e"}
Jul 13 21:30:44.656986 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:44.656818    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:44.658253 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:44.658229    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:30:44.672164 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:44.672119    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9" podStartSLOduration=1.168483891 podStartE2EDuration="4.672107965s" totalImagesPullingTime="3.503624074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:30:44.670889989 +0000 UTC m=+85.827283829" watchObservedRunningTime="2026-07-13 21:30:44.672107965 +0000 UTC m=+85.828501806"
Jul 13 21:30:57.597946 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:30:57.597911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:30:57.598405 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:30:57.598021    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:30:57.598405 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:30:57.598078    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls podName:7e46539f-e0bc-4eea-a0ca-e65633ccf553 nodeName:}" failed. No retries permitted until 2026-07-13 21:32:01.598065199 +0000 UTC m=+162.754459018 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls") pod "dns-default-tvf8f" (UID: "7e46539f-e0bc-4eea-a0ca-e65633ccf553") : secret "dns-default-metrics-tls" not found
Jul 13 21:31:01.622730 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:31:01.622697    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bkp9m"
Jul 13 21:31:04.498989 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:31:04.498952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:31:04.553706 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:31:04.553672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:31:29.002098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:31:29.002050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:31:29.002609 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:31:29.002240    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 13 21:31:29.002609 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:31:29.002347    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs podName:71c50b57-6bb8-4e0f-b2a9-1daab13a74cf nodeName:}" failed. No retries permitted until 2026-07-13 21:33:31.002328123 +0000 UTC m=+252.158721942 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs") pod "network-metrics-daemon-2njdj" (UID: "71c50b57-6bb8-4e0f-b2a9-1daab13a74cf") : secret "metrics-daemon-secret" not found
Jul 13 21:31:56.789343 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:31:56.789297    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tvf8f" podUID="7e46539f-e0bc-4eea-a0ca-e65633ccf553"
Jul 13 21:31:56.835234 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:31:56.835203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:31:57.376856 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:31:57.376822    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:32:01.612200 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:32:01.612129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:32:01.612622 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:32:01.612297    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 21:32:01.612622 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:32:01.612368    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls podName:7e46539f-e0bc-4eea-a0ca-e65633ccf553 nodeName:}" failed. No retries permitted until 2026-07-13 21:34:03.612350216 +0000 UTC m=+284.768744035 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls") pod "dns-default-tvf8f" (UID: "7e46539f-e0bc-4eea-a0ca-e65633ccf553") : secret "dns-default-metrics-tls" not found
Jul 13 21:32:11.358299 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:32:11.358203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:32:44.657967 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:32:44.657920    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9" podUID="626d24de-d296-4890-bf60-10c033ae84be" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Jul 13 21:32:44.946431 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:32:44.946396    2582 generic.go:356] "Generic (PLEG): container finished" podID="626d24de-d296-4890-bf60-10c033ae84be" containerID="e754014228aef97264c3815dea9293b910477ae3d076770780cf16475cd0b78e" exitCode=1
Jul 13 21:32:44.946595 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:32:44.946470    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9" event={"ID":"626d24de-d296-4890-bf60-10c033ae84be","Type":"ContainerDied","Data":"e754014228aef97264c3815dea9293b910477ae3d076770780cf16475cd0b78e"}
Jul 13 21:32:44.946779 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:32:44.946766    2582 scope.go:117] "RemoveContainer" containerID="e754014228aef97264c3815dea9293b910477ae3d076770780cf16475cd0b78e"
Jul 13 21:32:45.951039 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:32:45.950995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9" event={"ID":"626d24de-d296-4890-bf60-10c033ae84be","Type":"ContainerStarted","Data":"d69f3dfae9a408890304416cc624d50e78ad29d5a8571c1629944c2aee55d758"}
Jul 13 21:32:45.951597 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:32:45.951362    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:32:45.951948 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:32:45.951929    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5959874999-s9gv9"
Jul 13 21:33:13.600052 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:13.599999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:31.100223 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:31.100188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:33:31.100687 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:31.100301    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 13 21:33:31.100687 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:31.100358    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs podName:71c50b57-6bb8-4e0f-b2a9-1daab13a74cf nodeName:}" failed. No retries permitted until 2026-07-13 21:35:33.100344695 +0000 UTC m=+374.256738513 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs") pod "network-metrics-daemon-2njdj" (UID: "71c50b57-6bb8-4e0f-b2a9-1daab13a74cf") : secret "metrics-daemon-secret" not found
Jul 13 21:33:38.439519 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.439487    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56948c7b44-sgwgk"]
Jul 13 21:33:38.442504 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.442487    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-777b6c774d-p57vr"]
Jul 13 21:33:38.442614 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.442597    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.445098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.445075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.445302 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.445283    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 13 21:33:38.445384 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.445345    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9kttq\""
Jul 13 21:33:38.445590 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.445571    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 13 21:33:38.445629 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.445578    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 13 21:33:38.445702 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.445690    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 13 21:33:38.445807 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.445793    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 13 21:33:38.445882 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.445870    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 13 21:33:38.448635 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.448610    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 13 21:33:38.448776 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.448667    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 13 21:33:38.453793 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.453770    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 13 21:33:38.453898 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.453809    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rpb8r\""
Jul 13 21:33:38.457264 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.457241    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 13 21:33:38.457365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.457246    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56948c7b44-sgwgk"]
Jul 13 21:33:38.460727 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.460702    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-777b6c774d-p57vr"]
Jul 13 21:33:38.537291 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.537257    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-hx5cc"]
Jul 13 21:33:38.540381 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.540359    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-nslf5"]
Jul 13 21:33:38.540504 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.540488    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.542943 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.542921    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 21:33:38.543056 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.543023    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 13 21:33:38.543056 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.543043    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-v4txf\""
Jul 13 21:33:38.543169 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.543058    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 21:33:38.543220 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.543181    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.543352 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.543336    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 13 21:33:38.545536 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.545502    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 13 21:33:38.545536 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.545531    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bjvnh\""
Jul 13 21:33:38.545887 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.545861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.545963 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.545902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6bl\" (UniqueName: \"kubernetes.io/projected/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-kube-api-access-mj6bl\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.545963 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.545938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-installation-pull-secrets\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.545963 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.545944    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 13 21:33:38.546117 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.545970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmg4d\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-kube-api-access-pmg4d\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.546117 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.546117 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-bound-sa-token\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.546117 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-default-certificate\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.546377 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.546377 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546145    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-stats-auth\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.546377 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-image-registry-private-configuration\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.546517 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-certificates\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.546517 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-trusted-ca\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.546517 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546472    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 13 21:33:38.546517 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546486    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 13 21:33:38.546681 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.546491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/915298ab-6dc3-4743-8675-9ae3c60e6699-ca-trust-extracted\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.550788 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.550767    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 13 21:33:38.553009 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.552984    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 13 21:33:38.553589 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.553568    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-hx5cc"]
Jul 13 21:33:38.565757 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.565733    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-nslf5"]
Jul 13 21:33:38.647636 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-bound-sa-token\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.647636 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-default-certificate\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.647827 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fdf28c8-beba-416a-9165-66a689308580-serving-cert\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.647827 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46df3134-e1d1-42e7-aacc-ad226f9d84a6-config\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.647827 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fdf28c8-beba-416a-9165-66a689308580-tmp\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.647945 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-installation-pull-secrets\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.647945 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-trusted-ca\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.647945 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmg4d\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-kube-api-access-pmg4d\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.647945 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.647945 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-certificates\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.648198 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.647970    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/915298ab-6dc3-4743-8675-9ae3c60e6699-ca-trust-extracted\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.648198 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:38.648060    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle podName:6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:39.1480393 +0000 UTC m=+260.304433134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle") pod "router-default-56948c7b44-sgwgk" (UID: "6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6") : configmap references non-existent config key: service-ca.crt
Jul 13 21:33:38.648198 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.648198 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6bl\" (UniqueName: \"kubernetes.io/projected/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-kube-api-access-mj6bl\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.648198 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46df3134-e1d1-42e7-aacc-ad226f9d84a6-serving-cert\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.648460 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fdf28c8-beba-416a-9165-66a689308580-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.648460 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-image-registry-private-configuration\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.648460 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648310    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46df3134-e1d1-42e7-aacc-ad226f9d84a6-trusted-ca\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.648460 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:38.648227    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 21:33:38.648460 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:38.648400    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs podName:6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:39.148381558 +0000 UTC m=+260.304775389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs") pod "router-default-56948c7b44-sgwgk" (UID: "6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6") : secret "router-metrics-certs-default" not found
Jul 13 21:33:38.648460 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648403    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/915298ab-6dc3-4743-8675-9ae3c60e6699-ca-trust-extracted\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.648460 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5fdf28c8-beba-416a-9165-66a689308580-snapshots\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.648870 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-stats-auth\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.648870 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/5fdf28c8-beba-416a-9165-66a689308580-kube-api-access-hc4tw\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.648870 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkf4\" (UniqueName: \"kubernetes.io/projected/46df3134-e1d1-42e7-aacc-ad226f9d84a6-kube-api-access-fmkf4\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.648870 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.648870 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fdf28c8-beba-416a-9165-66a689308580-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.649076 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648873    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-trusted-ca\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.649076 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.648925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-certificates\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.649076 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:38.648938    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:33:38.649076 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:38.648948    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777b6c774d-p57vr: secret "image-registry-tls" not found
Jul 13 21:33:38.649076 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:38.649000    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls podName:915298ab-6dc3-4743-8675-9ae3c60e6699 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:39.148984822 +0000 UTC m=+260.305378642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls") pod "image-registry-777b6c774d-p57vr" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699") : secret "image-registry-tls" not found
Jul 13 21:33:38.650661 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.650631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-default-certificate\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.650793 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.650771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-image-registry-private-configuration\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.650834 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.650774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-installation-pull-secrets\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.650996 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.650982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-stats-auth\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.657562 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.657535    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmg4d\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-kube-api-access-pmg4d\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.657671 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.657632    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-bound-sa-token\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:38.658048 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.658028    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6bl\" (UniqueName: \"kubernetes.io/projected/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-kube-api-access-mj6bl\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:38.669782 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.669746    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:38.744526 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.744487    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:38.749447 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46df3134-e1d1-42e7-aacc-ad226f9d84a6-config\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.749515 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fdf28c8-beba-416a-9165-66a689308580-tmp\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.749637 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46df3134-e1d1-42e7-aacc-ad226f9d84a6-serving-cert\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.749713 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fdf28c8-beba-416a-9165-66a689308580-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.749768 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46df3134-e1d1-42e7-aacc-ad226f9d84a6-trusted-ca\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.749768 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5fdf28c8-beba-416a-9165-66a689308580-snapshots\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.749871 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/5fdf28c8-beba-416a-9165-66a689308580-kube-api-access-hc4tw\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.749871 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749821    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkf4\" (UniqueName: \"kubernetes.io/projected/46df3134-e1d1-42e7-aacc-ad226f9d84a6-kube-api-access-fmkf4\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.749972 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fdf28c8-beba-416a-9165-66a689308580-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.749972 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fdf28c8-beba-416a-9165-66a689308580-serving-cert\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.750248 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.750219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46df3134-e1d1-42e7-aacc-ad226f9d84a6-config\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.750366 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.749805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fdf28c8-beba-416a-9165-66a689308580-tmp\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.750366 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.750303    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fdf28c8-beba-416a-9165-66a689308580-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.750482 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.750435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5fdf28c8-beba-416a-9165-66a689308580-snapshots\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.750591 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.750566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46df3134-e1d1-42e7-aacc-ad226f9d84a6-trusted-ca\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.750857 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.750840    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fdf28c8-beba-416a-9165-66a689308580-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.752644 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.752614    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46df3134-e1d1-42e7-aacc-ad226f9d84a6-serving-cert\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.752722 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.752656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fdf28c8-beba-416a-9165-66a689308580-serving-cert\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.757654 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.757628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/5fdf28c8-beba-416a-9165-66a689308580-kube-api-access-hc4tw\") pod \"insights-operator-68d9dd94cf-hx5cc\" (UID: \"5fdf28c8-beba-416a-9165-66a689308580\") " pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.757793 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.757777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkf4\" (UniqueName: \"kubernetes.io/projected/46df3134-e1d1-42e7-aacc-ad226f9d84a6-kube-api-access-fmkf4\") pod \"console-operator-7866497494-nslf5\" (UID: \"46df3134-e1d1-42e7-aacc-ad226f9d84a6\") " pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.850580 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.850544    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc"
Jul 13 21:33:38.856984 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.856957    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:38.979598 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.979562    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-hx5cc"]
Jul 13 21:33:38.982379 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:33:38.982354    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdf28c8_beba_416a_9165_66a689308580.slice/crio-0eed9f325dacdc23a60620109f7a6e43e0ef7b14282bf65b773187876f80576e WatchSource:0}: Error finding container 0eed9f325dacdc23a60620109f7a6e43e0ef7b14282bf65b773187876f80576e: Status 404 returned error can't find the container with id 0eed9f325dacdc23a60620109f7a6e43e0ef7b14282bf65b773187876f80576e
Jul 13 21:33:38.996076 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:38.996016    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-nslf5"]
Jul 13 21:33:38.998946 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:33:38.998923    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46df3134_e1d1_42e7_aacc_ad226f9d84a6.slice/crio-a1fe3640c602e34ca58d336f938e833e26eda22a2479c305573c585c51406cd7 WatchSource:0}: Error finding container a1fe3640c602e34ca58d336f938e833e26eda22a2479c305573c585c51406cd7: Status 404 returned error can't find the container with id a1fe3640c602e34ca58d336f938e833e26eda22a2479c305573c585c51406cd7
Jul 13 21:33:39.099323 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:39.099289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc" event={"ID":"5fdf28c8-beba-416a-9165-66a689308580","Type":"ContainerStarted","Data":"0eed9f325dacdc23a60620109f7a6e43e0ef7b14282bf65b773187876f80576e"}
Jul 13 21:33:39.100259 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:39.100234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nslf5" event={"ID":"46df3134-e1d1-42e7-aacc-ad226f9d84a6","Type":"ContainerStarted","Data":"a1fe3640c602e34ca58d336f938e833e26eda22a2479c305573c585c51406cd7"}
Jul 13 21:33:39.153218 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:39.153178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:39.153369 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:39.153242    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:39.153369 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:39.153270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:39.153369 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:39.153330    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:33:39.153369 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:39.153347    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 21:33:39.153369 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:39.153350    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777b6c774d-p57vr: secret "image-registry-tls" not found
Jul 13 21:33:39.153546 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:39.153392    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs podName:6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:40.153377403 +0000 UTC m=+261.309771221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs") pod "router-default-56948c7b44-sgwgk" (UID: "6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6") : secret "router-metrics-certs-default" not found
Jul 13 21:33:39.153546 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:39.153404    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls podName:915298ab-6dc3-4743-8675-9ae3c60e6699 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:40.153398674 +0000 UTC m=+261.309792493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls") pod "image-registry-777b6c774d-p57vr" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699") : secret "image-registry-tls" not found
Jul 13 21:33:39.153546 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:39.153422    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle podName:6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:40.153408331 +0000 UTC m=+261.309802155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle") pod "router-default-56948c7b44-sgwgk" (UID: "6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6") : configmap references non-existent config key: service-ca.crt
Jul 13 21:33:40.162197 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:40.162138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:40.162624 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:40.162219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:40.162624 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:40.162316    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle podName:6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:42.162292577 +0000 UTC m=+263.318686404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle") pod "router-default-56948c7b44-sgwgk" (UID: "6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6") : configmap references non-existent config key: service-ca.crt
Jul 13 21:33:40.162624 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:40.162354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:40.162624 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:40.162378    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 21:33:40.162624 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:40.162438    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs podName:6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:42.162420686 +0000 UTC m=+263.318814510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs") pod "router-default-56948c7b44-sgwgk" (UID: "6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6") : secret "router-metrics-certs-default" not found
Jul 13 21:33:40.162624 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:40.162491    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:33:40.162624 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:40.162506    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777b6c774d-p57vr: secret "image-registry-tls" not found
Jul 13 21:33:40.162624 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:40.162546    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls podName:915298ab-6dc3-4743-8675-9ae3c60e6699 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:42.162533732 +0000 UTC m=+263.318927558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls") pod "image-registry-777b6c774d-p57vr" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699") : secret "image-registry-tls" not found
Jul 13 21:33:41.355769 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:41.355620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:41.454729 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:41.454698    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:42.111039 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:42.111003    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc" event={"ID":"5fdf28c8-beba-416a-9165-66a689308580","Type":"ContainerStarted","Data":"49ed613ae735e77ac414bb3ba0f160553e5e7fe6e080db6b579a4ce8da9455d4"}
Jul 13 21:33:42.112481 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:42.112462    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/0.log"
Jul 13 21:33:42.112606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:42.112493    2582 generic.go:356] "Generic (PLEG): container finished" podID="46df3134-e1d1-42e7-aacc-ad226f9d84a6" containerID="b58af6fc7d89f0ee98a38cbc22a65daa78efb3e3ecc06ebc546838ecad4a5a68" exitCode=255
Jul 13 21:33:42.112606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:42.112519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nslf5" event={"ID":"46df3134-e1d1-42e7-aacc-ad226f9d84a6","Type":"ContainerDied","Data":"b58af6fc7d89f0ee98a38cbc22a65daa78efb3e3ecc06ebc546838ecad4a5a68"}
Jul 13 21:33:42.112769 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:42.112754    2582 scope.go:117] "RemoveContainer" containerID="b58af6fc7d89f0ee98a38cbc22a65daa78efb3e3ecc06ebc546838ecad4a5a68"
Jul 13 21:33:42.128946 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:42.128907    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc" podStartSLOduration=1.143827014 podStartE2EDuration="4.128893125s" totalImagesPullingTime="2.985066111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:33:42.127960244 +0000 UTC m=+263.284354084" watchObservedRunningTime="2026-07-13 21:33:42.128893125 +0000 UTC m=+263.285287002"
Jul 13 21:33:42.176792 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:42.176764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:42.176902 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:42.176814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:42.176902 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:42.176834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:42.177012 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:42.176922    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 21:33:42.177012 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:42.176929    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:33:42.177012 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:42.176951    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777b6c774d-p57vr: secret "image-registry-tls" not found
Jul 13 21:33:42.177012 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:42.176964    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle podName:6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:46.176941373 +0000 UTC m=+267.333335196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle") pod "router-default-56948c7b44-sgwgk" (UID: "6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6") : configmap references non-existent config key: service-ca.crt
Jul 13 21:33:42.177012 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:42.176997    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls podName:915298ab-6dc3-4743-8675-9ae3c60e6699 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:46.176979832 +0000 UTC m=+267.333373658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls") pod "image-registry-777b6c774d-p57vr" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699") : secret "image-registry-tls" not found
Jul 13 21:33:42.177012 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:42.177014    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs podName:6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:46.177005138 +0000 UTC m=+267.333398961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs") pod "router-default-56948c7b44-sgwgk" (UID: "6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6") : secret "router-metrics-certs-default" not found
Jul 13 21:33:43.117089 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.117063    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/1.log"
Jul 13 21:33:43.117545 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.117496    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/0.log"
Jul 13 21:33:43.117545 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.117536    2582 generic.go:356] "Generic (PLEG): container finished" podID="46df3134-e1d1-42e7-aacc-ad226f9d84a6" containerID="a52a675bf4e5b5527bf93d9ee570fd374bb8311668981b8ce77d8be1fdef3c59" exitCode=255
Jul 13 21:33:43.117653 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.117621    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nslf5" event={"ID":"46df3134-e1d1-42e7-aacc-ad226f9d84a6","Type":"ContainerDied","Data":"a52a675bf4e5b5527bf93d9ee570fd374bb8311668981b8ce77d8be1fdef3c59"}
Jul 13 21:33:43.117708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.117679    2582 scope.go:117] "RemoveContainer" containerID="b58af6fc7d89f0ee98a38cbc22a65daa78efb3e3ecc06ebc546838ecad4a5a68"
Jul 13 21:33:43.118006 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.117989    2582 scope.go:117] "RemoveContainer" containerID="a52a675bf4e5b5527bf93d9ee570fd374bb8311668981b8ce77d8be1fdef3c59"
Jul 13 21:33:43.118244 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:43.118220    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-nslf5_openshift-console-operator(46df3134-e1d1-42e7-aacc-ad226f9d84a6)\"" pod="openshift-console-operator/console-operator-7866497494-nslf5" podUID="46df3134-e1d1-42e7-aacc-ad226f9d84a6"
Jul 13 21:33:43.228787 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.228752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:43.340364 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.340325    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:43.429165 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.429041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:43.528923 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.528872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:43.658110 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.658076    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:43.712668 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.712580    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:43.795283 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.795247    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-79grf"]
Jul 13 21:33:43.800232 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.800211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.803237 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.803217    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 21:33:43.803355 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.803217    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 21:33:43.803355 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.803225    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tf6cc\""
Jul 13 21:33:43.811610 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.811590    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-79grf"]
Jul 13 21:33:43.830898 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.830872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:43.889533 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.889501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.889533 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.889535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p2qv\" (UniqueName: \"kubernetes.io/projected/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-api-access-6p2qv\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.889760 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.889600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f5a438a-b79a-4fe8-9621-6ec09a318089-data-volume\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.889760 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.889661    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.889760 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.889694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f5a438a-b79a-4fe8-9621-6ec09a318089-crio-socket\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.890083 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.890062    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:43.990022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.989988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.990219 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.990028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p2qv\" (UniqueName: \"kubernetes.io/projected/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-api-access-6p2qv\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.990219 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:43.990109    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 21:33:43.990219 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:43.990181    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm podName:7f5a438a-b79a-4fe8-9621-6ec09a318089 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:44.490166178 +0000 UTC m=+265.646560010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm") pod "insights-runtime-extractor-79grf" (UID: "7f5a438a-b79a-4fe8-9621-6ec09a318089") : configmap "kube-rbac-proxy" not found
Jul 13 21:33:43.990408 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.990244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f5a438a-b79a-4fe8-9621-6ec09a318089-data-volume\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.990408 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.990326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.990408 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.990369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f5a438a-b79a-4fe8-9621-6ec09a318089-crio-socket\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.990595 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:43.990471    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 21:33:43.990595 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.990488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f5a438a-b79a-4fe8-9621-6ec09a318089-crio-socket\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.990595 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:43.990522    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls podName:7f5a438a-b79a-4fe8-9621-6ec09a318089 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:44.49050458 +0000 UTC m=+265.646898405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls") pod "insights-runtime-extractor-79grf" (UID: "7f5a438a-b79a-4fe8-9621-6ec09a318089") : secret "insights-runtime-extractor-tls" not found
Jul 13 21:33:43.990778 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.990626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f5a438a-b79a-4fe8-9621-6ec09a318089-data-volume\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:43.990778 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.990748    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:43.998574 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:43.998555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p2qv\" (UniqueName: \"kubernetes.io/projected/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-api-access-6p2qv\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:44.121437 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:44.121406    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/1.log"
Jul 13 21:33:44.121809 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:44.121718    2582 scope.go:117] "RemoveContainer" containerID="a52a675bf4e5b5527bf93d9ee570fd374bb8311668981b8ce77d8be1fdef3c59"
Jul 13 21:33:44.121873 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:44.121855    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-nslf5_openshift-console-operator(46df3134-e1d1-42e7-aacc-ad226f9d84a6)\"" pod="openshift-console-operator/console-operator-7866497494-nslf5" podUID="46df3134-e1d1-42e7-aacc-ad226f9d84a6"
Jul 13 21:33:44.494308 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:44.494275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:44.494484 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:44.494365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:44.494484 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:44.494383    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 21:33:44.494484 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:44.494443    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm podName:7f5a438a-b79a-4fe8-9621-6ec09a318089 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:45.494425626 +0000 UTC m=+266.650819445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm") pod "insights-runtime-extractor-79grf" (UID: "7f5a438a-b79a-4fe8-9621-6ec09a318089") : configmap "kube-rbac-proxy" not found
Jul 13 21:33:44.494673 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:44.494500    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 21:33:44.494673 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:44.494551    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls podName:7f5a438a-b79a-4fe8-9621-6ec09a318089 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:45.494539109 +0000 UTC m=+266.650932933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls") pod "insights-runtime-extractor-79grf" (UID: "7f5a438a-b79a-4fe8-9621-6ec09a318089") : secret "insights-runtime-extractor-tls" not found
Jul 13 21:33:45.502033 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:45.501974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:45.502439 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:45.502064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:45.502439 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:45.502106    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 21:33:45.502439 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:45.502172    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 21:33:45.502439 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:45.502194    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm podName:7f5a438a-b79a-4fe8-9621-6ec09a318089 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:47.502176672 +0000 UTC m=+268.658570509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm") pod "insights-runtime-extractor-79grf" (UID: "7f5a438a-b79a-4fe8-9621-6ec09a318089") : configmap "kube-rbac-proxy" not found
Jul 13 21:33:45.502439 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:45.502224    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls podName:7f5a438a-b79a-4fe8-9621-6ec09a318089 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:47.502212063 +0000 UTC m=+268.658605882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls") pod "insights-runtime-extractor-79grf" (UID: "7f5a438a-b79a-4fe8-9621-6ec09a318089") : secret "insights-runtime-extractor-tls" not found
Jul 13 21:33:45.574082 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:45.574037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:45.622495 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:45.622462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:46.207628 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:46.207585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:46.207628 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:46.207634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:46.207857 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:46.207726    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:46.207857 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:46.207767    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle podName:6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:54.207747333 +0000 UTC m=+275.364141155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle") pod "router-default-56948c7b44-sgwgk" (UID: "6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6") : configmap references non-existent config key: service-ca.crt
Jul 13 21:33:46.207857 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:46.207825    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 21:33:46.207857 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:46.207839    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777b6c774d-p57vr: secret "image-registry-tls" not found
Jul 13 21:33:46.207998 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:46.207875    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls podName:915298ab-6dc3-4743-8675-9ae3c60e6699 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:54.207863966 +0000 UTC m=+275.364257790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls") pod "image-registry-777b6c774d-p57vr" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699") : secret "image-registry-tls" not found
Jul 13 21:33:46.207998 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:46.207824    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 21:33:46.207998 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:46.207914    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs podName:6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:54.207904208 +0000 UTC m=+275.364298035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs") pod "router-default-56948c7b44-sgwgk" (UID: "6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6") : secret "router-metrics-certs-default" not found
Jul 13 21:33:46.415201 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:46.415166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:46.589407 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:46.589379    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rh74n_4473539f-ed4b-437c-9814-8ee29ac74bf6/dns-node-resolver/0.log"
Jul 13 21:33:47.254503 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:47.254463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:47.356266 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:47.356236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:47.499139 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:47.499093    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:47.516935 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:47.516865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:47.517074 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:47.516944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:47.517074 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:47.517002    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 21:33:47.517074 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:47.517021    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 21:33:47.517074 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:47.517060    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls podName:7f5a438a-b79a-4fe8-9621-6ec09a318089 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:51.517043771 +0000 UTC m=+272.673437591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls") pod "insights-runtime-extractor-79grf" (UID: "7f5a438a-b79a-4fe8-9621-6ec09a318089") : secret "insights-runtime-extractor-tls" not found
Jul 13 21:33:47.517074 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:47.517073    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm podName:7f5a438a-b79a-4fe8-9621-6ec09a318089 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:51.517067477 +0000 UTC m=+272.673461296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm") pod "insights-runtime-extractor-79grf" (UID: "7f5a438a-b79a-4fe8-9621-6ec09a318089") : configmap "kube-rbac-proxy" not found
Jul 13 21:33:47.599393 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:47.599355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:47.788923 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:47.788852    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tbqlb_721b2fc5-6fe6-49a2-abaf-ea836e1d93bb/node-ca/0.log"
Jul 13 21:33:47.917004 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:47.916964    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:48.017945 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:48.017913    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:48.074315 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:48.074217    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:48.189044 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:48.189012    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9znbk_f3d9f579-5163-4bf3-906e-e497ca6d4d89/serve-healthcheck-canary/0.log"
Jul 13 21:33:48.751475 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:48.751440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:48.857549 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:48.857518    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:48.857673 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:48.857568    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:33:48.857865 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:48.857854    2582 scope.go:117] "RemoveContainer" containerID="a52a675bf4e5b5527bf93d9ee570fd374bb8311668981b8ce77d8be1fdef3c59"
Jul 13 21:33:48.858007 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:48.857989    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-nslf5_openshift-console-operator(46df3134-e1d1-42e7-aacc-ad226f9d84a6)\"" pod="openshift-console-operator/console-operator-7866497494-nslf5" podUID="46df3134-e1d1-42e7-aacc-ad226f9d84a6"
Jul 13 21:33:49.353936 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:49.353899    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:49.627411 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:49.627318    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:49.780636 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:49.780591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:49.957811 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:49.957718    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:50.183806 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:50.183780    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:50.747115 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:50.747081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:51.146410 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:51.146322    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:51.205765 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:51.205737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:51.550615 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:51.550590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:51.550760 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:51.550642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:51.550760 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:51.550717    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 21:33:51.550760 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:51.550740    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 21:33:51.550856 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:51.550783    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm podName:7f5a438a-b79a-4fe8-9621-6ec09a318089 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:59.550766983 +0000 UTC m=+280.707160807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm") pod "insights-runtime-extractor-79grf" (UID: "7f5a438a-b79a-4fe8-9621-6ec09a318089") : configmap "kube-rbac-proxy" not found
Jul 13 21:33:51.550856 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:51.550799    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls podName:7f5a438a-b79a-4fe8-9621-6ec09a318089 nodeName:}" failed. No retries permitted until 2026-07-13 21:33:59.55079285 +0000 UTC m=+280.707186669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls") pod "insights-runtime-extractor-79grf" (UID: "7f5a438a-b79a-4fe8-9621-6ec09a318089") : secret "insights-runtime-extractor-tls" not found
Jul 13 21:33:51.554622 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:51.554598    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:51.655919 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:51.655896    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:51.756666 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:51.756636    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:52.357075 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:52.357042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:52.697543 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:52.697456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:52.951946 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:52.951879    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:53.151524 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:53.151489    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:53.546506 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:53.546481    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:53.750887 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:53.750850    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:53.852028 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:53.851957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:53.953691 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:53.953651    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:54.271944 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.271915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:54.272119 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.271980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:54.272119 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.272007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:54.272646 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.272627    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-service-ca-bundle\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:54.274382 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.274361    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6-metrics-certs\") pod \"router-default-56948c7b44-sgwgk\" (UID: \"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6\") " pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:54.274459 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.274391    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls\") pod \"image-registry-777b6c774d-p57vr\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") " pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:54.356726 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.356703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:54.363606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.363588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:54.493857 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.493820    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56948c7b44-sgwgk"]
Jul 13 21:33:54.496520 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:33:54.496495    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8a2d79_ce7e_4eb4_b623_bbe7428c36d6.slice/crio-1068f1e37e5a9458c984f6b5ae44c0598b969e1a8c24b57d54b1a1f048366973 WatchSource:0}: Error finding container 1068f1e37e5a9458c984f6b5ae44c0598b969e1a8c24b57d54b1a1f048366973: Status 404 returned error can't find the container with id 1068f1e37e5a9458c984f6b5ae44c0598b969e1a8c24b57d54b1a1f048366973
Jul 13 21:33:54.508875 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.508851    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-777b6c774d-p57vr"]
Jul 13 21:33:54.511344 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:33:54.511306    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915298ab_6dc3_4743_8675_9ae3c60e6699.slice/crio-5772960066fa51e5fe641f7bcbe08d70da41ed22b0f986d8a424f7838e8933ad WatchSource:0}: Error finding container 5772960066fa51e5fe641f7bcbe08d70da41ed22b0f986d8a424f7838e8933ad: Status 404 returned error can't find the container with id 5772960066fa51e5fe641f7bcbe08d70da41ed22b0f986d8a424f7838e8933ad
Jul 13 21:33:54.950844 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:54.950772    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:55.156178 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.156125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56948c7b44-sgwgk" event={"ID":"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6","Type":"ContainerStarted","Data":"7b9db6611f5f16f50ee5f6ceff86ab7c7cd0e02cc6062c9393bafb6f339d0877"}
Jul 13 21:33:55.156376 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.156219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56948c7b44-sgwgk" event={"ID":"6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6","Type":"ContainerStarted","Data":"1068f1e37e5a9458c984f6b5ae44c0598b969e1a8c24b57d54b1a1f048366973"}
Jul 13 21:33:55.157538 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.157513    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777b6c774d-p57vr" event={"ID":"915298ab-6dc3-4743-8675-9ae3c60e6699","Type":"ContainerStarted","Data":"fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7"}
Jul 13 21:33:55.157644 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.157552    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777b6c774d-p57vr" event={"ID":"915298ab-6dc3-4743-8675-9ae3c60e6699","Type":"ContainerStarted","Data":"5772960066fa51e5fe641f7bcbe08d70da41ed22b0f986d8a424f7838e8933ad"}
Jul 13 21:33:55.157688 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.157642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:33:55.173191 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.173136    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-56948c7b44-sgwgk" podStartSLOduration=17.173124246 podStartE2EDuration="17.173124246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:33:55.173097866 +0000 UTC m=+276.329491706" watchObservedRunningTime="2026-07-13 21:33:55.173124246 +0000 UTC m=+276.329518087"
Jul 13 21:33:55.190872 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.190836    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-777b6c774d-p57vr" podStartSLOduration=17.190824678 podStartE2EDuration="17.190824678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:33:55.189966327 +0000 UTC m=+276.346360159" watchObservedRunningTime="2026-07-13 21:33:55.190824678 +0000 UTC m=+276.347218520"
Jul 13 21:33:55.349021 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.348983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:55.361264 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.361235    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:55.361389 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.361308    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:55.552131 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.552097    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:55.747333 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.747299    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:55.951635 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:55.951592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:56.161126 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:56.161045    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:56.162251 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:56.162230    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56948c7b44-sgwgk"
Jul 13 21:33:56.554108 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:56.554079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:56.754788 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:56.754758    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:57.146234 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:57.146196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:57.349736 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:57.349705    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:57.751115 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:57.751080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:57.951932 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:57.951898    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:33:59.612109 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:59.612078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:59.612476 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:59.612136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:59.612683 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:59.612663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f5a438a-b79a-4fe8-9621-6ec09a318089-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:59.614352 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:59.614336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f5a438a-b79a-4fe8-9621-6ec09a318089-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79grf\" (UID: \"7f5a438a-b79a-4fe8-9621-6ec09a318089\") " pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:59.709078 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:59.709050    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-79grf"
Jul 13 21:33:59.833062 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:33:59.833033    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-79grf"]
Jul 13 21:33:59.839430 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:33:59.839390    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tvf8f" podUID="7e46539f-e0bc-4eea-a0ca-e65633ccf553"
Jul 13 21:33:59.840787 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:33:59.840756    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5a438a_b79a_4fe8_9621_6ec09a318089.slice/crio-fcd8edcc738b690fc13f27097e417cf11a3451d402d5beb46dd32381bdd8759e WatchSource:0}: Error finding container fcd8edcc738b690fc13f27097e417cf11a3451d402d5beb46dd32381bdd8759e: Status 404 returned error can't find the container with id fcd8edcc738b690fc13f27097e417cf11a3451d402d5beb46dd32381bdd8759e
Jul 13 21:34:00.173402 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:00.173376    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:34:00.173563 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:00.173374    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79grf" event={"ID":"7f5a438a-b79a-4fe8-9621-6ec09a318089","Type":"ContainerStarted","Data":"fcd8edcc738b690fc13f27097e417cf11a3451d402d5beb46dd32381bdd8759e"}
Jul 13 21:34:00.358887 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:00.358516    2582 scope.go:117] "RemoveContainer" containerID="a52a675bf4e5b5527bf93d9ee570fd374bb8311668981b8ce77d8be1fdef3c59"
Jul 13 21:34:01.178581 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:01.178552    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:34:01.179033 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:01.179011    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/1.log"
Jul 13 21:34:01.179080 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:01.179057    2582 generic.go:356] "Generic (PLEG): container finished" podID="46df3134-e1d1-42e7-aacc-ad226f9d84a6" containerID="20ed6d1b07de7ed5b9fb7167f2bfe448dcd5ff38f39cb8170652141b36a7b734" exitCode=255
Jul 13 21:34:01.179131 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:01.179112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nslf5" event={"ID":"46df3134-e1d1-42e7-aacc-ad226f9d84a6","Type":"ContainerDied","Data":"20ed6d1b07de7ed5b9fb7167f2bfe448dcd5ff38f39cb8170652141b36a7b734"}
Jul 13 21:34:01.179217 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:01.179179    2582 scope.go:117] "RemoveContainer" containerID="a52a675bf4e5b5527bf93d9ee570fd374bb8311668981b8ce77d8be1fdef3c59"
Jul 13 21:34:01.179607 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:01.179587    2582 scope.go:117] "RemoveContainer" containerID="20ed6d1b07de7ed5b9fb7167f2bfe448dcd5ff38f39cb8170652141b36a7b734"
Jul 13 21:34:01.179795 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:34:01.179774    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-nslf5_openshift-console-operator(46df3134-e1d1-42e7-aacc-ad226f9d84a6)\"" pod="openshift-console-operator/console-operator-7866497494-nslf5" podUID="46df3134-e1d1-42e7-aacc-ad226f9d84a6"
Jul 13 21:34:02.184599 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:02.184563    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:34:03.640504 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:03.640474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:34:03.642690 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:03.642670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e46539f-e0bc-4eea-a0ca-e65633ccf553-metrics-tls\") pod \"dns-default-tvf8f\" (UID: \"7e46539f-e0bc-4eea-a0ca-e65633ccf553\") " pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:34:03.776602 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:03.776572    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l6v89\""
Jul 13 21:34:03.784675 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:03.784643    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:34:03.906376 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:03.906347    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tvf8f"]
Jul 13 21:34:03.908997 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:34:03.908967    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e46539f_e0bc_4eea_a0ca_e65633ccf553.slice/crio-50741125f270edd4f1f0049fd82695fb8ccd38fea8a217c3c7e91cc3c9ee3715 WatchSource:0}: Error finding container 50741125f270edd4f1f0049fd82695fb8ccd38fea8a217c3c7e91cc3c9ee3715: Status 404 returned error can't find the container with id 50741125f270edd4f1f0049fd82695fb8ccd38fea8a217c3c7e91cc3c9ee3715
Jul 13 21:34:04.191502 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:04.191469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tvf8f" event={"ID":"7e46539f-e0bc-4eea-a0ca-e65633ccf553","Type":"ContainerStarted","Data":"50741125f270edd4f1f0049fd82695fb8ccd38fea8a217c3c7e91cc3c9ee3715"}
Jul 13 21:34:04.192793 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:04.192767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79grf" event={"ID":"7f5a438a-b79a-4fe8-9621-6ec09a318089","Type":"ContainerStarted","Data":"2dc6d7bf547f242607dbd9c05655e2e17ca8ee8eaa3eb7bf5810f7e5f87dcea5"}
Jul 13 21:34:05.198802 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:05.198764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79grf" event={"ID":"7f5a438a-b79a-4fe8-9621-6ec09a318089","Type":"ContainerStarted","Data":"0ea99d844c096d359bd6bae900564464a99ce0fde974bdd40d1050469935b3f5"}
Jul 13 21:34:05.554709 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:05.554635    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:05.609641 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:05.609591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:07.207238 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.207208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79grf" event={"ID":"7f5a438a-b79a-4fe8-9621-6ec09a318089","Type":"ContainerStarted","Data":"e2de094511c04ae594fc1e0f170b11e4e426785a00e46190efda37b8391899a1"}
Jul 13 21:34:07.208554 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.208532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tvf8f" event={"ID":"7e46539f-e0bc-4eea-a0ca-e65633ccf553","Type":"ContainerStarted","Data":"34c112798ce6a0684592df8acea7135c64eefc2b9da1183dc4c3914f4af25dff"}
Jul 13 21:34:07.225141 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.225096    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-79grf" podStartSLOduration=17.091415599 podStartE2EDuration="24.22508088s" totalImagesPullingTime="7.133665281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:33:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:07.223373265 +0000 UTC m=+288.379767336" watchObservedRunningTime="2026-07-13 21:34:07.22508088 +0000 UTC m=+288.381474722"
Jul 13 21:34:07.602604 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.602574    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-777b6c774d-p57vr"]
Jul 13 21:34:07.637546 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.637519    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c7b7959d5-wvnrk"]
Jul 13 21:34:07.640349 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.640331    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.653117 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.653098    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c7b7959d5-wvnrk"]
Jul 13 21:34:07.667674 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.667654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:07.670666 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.670647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d23ddb8-f815-405d-beb7-e0e83108a305-image-registry-private-configuration\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.670728 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.670675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d23ddb8-f815-405d-beb7-e0e83108a305-ca-trust-extracted\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.670728 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.670699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d23ddb8-f815-405d-beb7-e0e83108a305-trusted-ca\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.670728 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.670721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d23ddb8-f815-405d-beb7-e0e83108a305-registry-tls\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.670834 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.670769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d23ddb8-f815-405d-beb7-e0e83108a305-bound-sa-token\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.670834 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.670793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d23ddb8-f815-405d-beb7-e0e83108a305-installation-pull-secrets\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.670920 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.670841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d23ddb8-f815-405d-beb7-e0e83108a305-registry-certificates\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.670920 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.670872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xts8m\" (UniqueName: \"kubernetes.io/projected/9d23ddb8-f815-405d-beb7-e0e83108a305-kube-api-access-xts8m\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.766115 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.766083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:07.772114 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.772090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d23ddb8-f815-405d-beb7-e0e83108a305-image-registry-private-configuration\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.772182 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.772128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d23ddb8-f815-405d-beb7-e0e83108a305-ca-trust-extracted\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.772182 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.772177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d23ddb8-f815-405d-beb7-e0e83108a305-trusted-ca\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.772256 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.772203    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d23ddb8-f815-405d-beb7-e0e83108a305-registry-tls\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.772351 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.772331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d23ddb8-f815-405d-beb7-e0e83108a305-bound-sa-token\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.772427 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.772384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d23ddb8-f815-405d-beb7-e0e83108a305-installation-pull-secrets\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.772482 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.772425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d23ddb8-f815-405d-beb7-e0e83108a305-registry-certificates\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.772482 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.772458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xts8m\" (UniqueName: \"kubernetes.io/projected/9d23ddb8-f815-405d-beb7-e0e83108a305-kube-api-access-xts8m\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.772759 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.772593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d23ddb8-f815-405d-beb7-e0e83108a305-ca-trust-extracted\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.773260 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.773235    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d23ddb8-f815-405d-beb7-e0e83108a305-registry-certificates\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.773412 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.773392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d23ddb8-f815-405d-beb7-e0e83108a305-trusted-ca\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.774766 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.774750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d23ddb8-f815-405d-beb7-e0e83108a305-image-registry-private-configuration\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.775104 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.775089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d23ddb8-f815-405d-beb7-e0e83108a305-registry-tls\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.775262 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.775240    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d23ddb8-f815-405d-beb7-e0e83108a305-installation-pull-secrets\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.780967 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.780943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xts8m\" (UniqueName: \"kubernetes.io/projected/9d23ddb8-f815-405d-beb7-e0e83108a305-kube-api-access-xts8m\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.781410 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.781394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d23ddb8-f815-405d-beb7-e0e83108a305-bound-sa-token\") pod \"image-registry-7c7b7959d5-wvnrk\" (UID: \"9d23ddb8-f815-405d-beb7-e0e83108a305\") " pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:07.866684 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.866592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:07.949117 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:07.949079    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:08.076101 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.076067    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c7b7959d5-wvnrk"]
Jul 13 21:34:08.079365 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:34:08.079336    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d23ddb8_f815_405d_beb7_e0e83108a305.slice/crio-90262dbf569e60614703b935686c181e90894887dd852feb5aef4a6d3b5f9d45 WatchSource:0}: Error finding container 90262dbf569e60614703b935686c181e90894887dd852feb5aef4a6d3b5f9d45: Status 404 returned error can't find the container with id 90262dbf569e60614703b935686c181e90894887dd852feb5aef4a6d3b5f9d45
Jul 13 21:34:08.213412 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.213379    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tvf8f" event={"ID":"7e46539f-e0bc-4eea-a0ca-e65633ccf553","Type":"ContainerStarted","Data":"49965cc54f8fd06112237c6f9ddd7c7e2b4558406ba939108ca753b55ea5daad"}
Jul 13 21:34:08.213871 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.213510    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:34:08.214912 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.214878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk" event={"ID":"9d23ddb8-f815-405d-beb7-e0e83108a305","Type":"ContainerStarted","Data":"3da340501d4a4cdfc2bb05556b799b8503f41ed0f429675bc5d34508e13be5af"}
Jul 13 21:34:08.215005 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.214935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk" event={"ID":"9d23ddb8-f815-405d-beb7-e0e83108a305","Type":"ContainerStarted","Data":"90262dbf569e60614703b935686c181e90894887dd852feb5aef4a6d3b5f9d45"}
Jul 13 21:34:08.215077 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.215058    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:08.232215 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.232141    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tvf8f" podStartSLOduration=252.069214983 podStartE2EDuration="4m15.23211951s" totalImagesPullingTime="3.162904527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:08.231741944 +0000 UTC m=+289.388135786" watchObservedRunningTime="2026-07-13 21:34:08.23211951 +0000 UTC m=+289.388513352"
Jul 13 21:34:08.249533 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.249450    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk" podStartSLOduration=1.249430301 podStartE2EDuration="1.249430301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:08.248967265 +0000 UTC m=+289.405361108" watchObservedRunningTime="2026-07-13 21:34:08.249430301 +0000 UTC m=+289.405824143"
Jul 13 21:34:08.857453 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.857419    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:34:08.857614 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.857470    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:34:08.857778 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:08.857765    2582 scope.go:117] "RemoveContainer" containerID="20ed6d1b07de7ed5b9fb7167f2bfe448dcd5ff38f39cb8170652141b36a7b734"
Jul 13 21:34:08.857947 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:34:08.857928    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-nslf5_openshift-console-operator(46df3134-e1d1-42e7-aacc-ad226f9d84a6)\"" pod="openshift-console-operator/console-operator-7866497494-nslf5" podUID="46df3134-e1d1-42e7-aacc-ad226f9d84a6"
Jul 13 21:34:13.057378 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:13.057345    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:13.158718 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:13.158684    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:13.231291 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:13.231255    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:13.515955 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:13.515925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:14.359552 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:34:14.359517    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2njdj" podUID="71c50b57-6bb8-4e0f-b2a9-1daab13a74cf"
Jul 13 21:34:17.234242 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:17.234207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:17.295538 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:17.295509    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:17.608315 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:17.608235    2582 patch_prober.go:28] interesting pod/image-registry-777b6c774d-p57vr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 13 21:34:17.608315 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:17.608282    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-777b6c774d-p57vr" podUID="915298ab-6dc3-4743-8675-9ae3c60e6699" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 21:34:17.980590 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:17.980562    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:18.221579 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:18.221554    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tvf8f"
Jul 13 21:34:19.193687 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:19.193664    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:34:19.196193 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:19.196172    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:34:19.198470 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:19.198449    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:34:19.200954 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:19.200936    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:34:19.204405 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:19.204390    2582 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 21:34:20.786078 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:20.786047    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:21.358614 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:21.358585    2582 scope.go:117] "RemoveContainer" containerID="20ed6d1b07de7ed5b9fb7167f2bfe448dcd5ff38f39cb8170652141b36a7b734"
Jul 13 21:34:22.137509 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:22.137474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:22.263118 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:22.263086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:22.264329 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:22.264310    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:34:22.264415 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:22.264374    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nslf5" event={"ID":"46df3134-e1d1-42e7-aacc-ad226f9d84a6","Type":"ContainerStarted","Data":"b5dc10b673777b4b80643a4de65651aca10bc299092f755f7f7067478ca65bed"}
Jul 13 21:34:22.264663 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:22.264625    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:34:22.269498 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:22.269477    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-nslf5"
Jul 13 21:34:22.281127 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:22.281090    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-nslf5" podStartSLOduration=41.309994401 podStartE2EDuration="44.281078469s" totalImagesPullingTime="2.971084068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:22.280326418 +0000 UTC m=+303.436720260" watchObservedRunningTime="2026-07-13 21:34:22.281078469 +0000 UTC m=+303.437472310"
Jul 13 21:34:22.641090 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:22.641057    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:22.690901 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:22.690872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:22.844686 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:22.844648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:23.037236 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.037196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:23.086231 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.086200    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:23.366554 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.366470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:23.465520 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.465495    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:23.572285 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.572257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:23.634979 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.634901    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xx6xg"]
Jul 13 21:34:23.637042 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.637020    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.639650 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.639631    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vvh8d\""
Jul 13 21:34:23.639760 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.639631    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 21:34:23.639760 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.639683    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 21:34:23.640699 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.640679    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 21:34:23.640791 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.640746    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 21:34:23.640791 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.640766    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 21:34:23.640905 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.640748    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 21:34:23.666851 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.666829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:23.687817 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.687793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.687910 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.687821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hms8\" (UniqueName: \"kubernetes.io/projected/c2fd3764-1986-4104-98ec-58a05d107526-kube-api-access-9hms8\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.687910 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.687843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-textfile\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.688005 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.687904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2fd3764-1986-4104-98ec-58a05d107526-metrics-client-ca\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.688005 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.687954    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-wtmp\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.688005 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.687974    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-tls\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.688005 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.687993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2fd3764-1986-4104-98ec-58a05d107526-root\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.688228 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.688008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2fd3764-1986-4104-98ec-58a05d107526-sys\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.688228 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.688097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-accelerators-collector-config\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.728546 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.728523    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:23.788920 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.788900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-accelerators-collector-config\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.788935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.788952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hms8\" (UniqueName: \"kubernetes.io/projected/c2fd3764-1986-4104-98ec-58a05d107526-kube-api-access-9hms8\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.788977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-textfile\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2fd3764-1986-4104-98ec-58a05d107526-metrics-client-ca\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789216 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-wtmp\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789216 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789102    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-tls\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789216 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2fd3764-1986-4104-98ec-58a05d107526-root\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789216 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2fd3764-1986-4104-98ec-58a05d107526-sys\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789401 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2fd3764-1986-4104-98ec-58a05d107526-sys\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789401 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2fd3764-1986-4104-98ec-58a05d107526-root\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789493 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-wtmp\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789646 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789624    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-accelerators-collector-config\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789646 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789641    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-textfile\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.789883 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.789862    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2fd3764-1986-4104-98ec-58a05d107526-metrics-client-ca\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.791431 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.791413    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.791585 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.791564    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2fd3764-1986-4104-98ec-58a05d107526-node-exporter-tls\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.796367 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.796345    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hms8\" (UniqueName: \"kubernetes.io/projected/c2fd3764-1986-4104-98ec-58a05d107526-kube-api-access-9hms8\") pod \"node-exporter-xx6xg\" (UID: \"c2fd3764-1986-4104-98ec-58a05d107526\") " pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.946427 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.946357    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xx6xg"
Jul 13 21:34:23.956611 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:34:23.956584    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fd3764_1986_4104_98ec_58a05d107526.slice/crio-d037c6e26477caa56d670789d0402965acf10fa7be4f088b14364c7a63ea533b WatchSource:0}: Error finding container d037c6e26477caa56d670789d0402965acf10fa7be4f088b14364c7a63ea533b: Status 404 returned error can't find the container with id d037c6e26477caa56d670789d0402965acf10fa7be4f088b14364c7a63ea533b
Jul 13 21:34:23.958674 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.958646    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:34:23.962562 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:23.962535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:24.062096 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:24.062058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:24.118467 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:24.118434    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:24.271783 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:24.271750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xx6xg" event={"ID":"c2fd3764-1986-4104-98ec-58a05d107526","Type":"ContainerStarted","Data":"d037c6e26477caa56d670789d0402965acf10fa7be4f088b14364c7a63ea533b"}
Jul 13 21:34:24.439694 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:24.439666    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:24.586247 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:24.586140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:24.667613 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:24.667565    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:24.988458 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:24.988433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:25.089403 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.089214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:25.140622 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.140596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:25.275910 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.275887    2582 generic.go:356] "Generic (PLEG): container finished" podID="c2fd3764-1986-4104-98ec-58a05d107526" containerID="ee06b731c68ef08f6966189b30fcaf3122f6a5e05c762f350736531e69596603" exitCode=0
Jul 13 21:34:25.276020 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.275928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xx6xg" event={"ID":"c2fd3764-1986-4104-98ec-58a05d107526","Type":"ContainerDied","Data":"ee06b731c68ef08f6966189b30fcaf3122f6a5e05c762f350736531e69596603"}
Jul 13 21:34:25.575289 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.575202    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:25.593040 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.593010    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"]
Jul 13 21:34:25.595259 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.595241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.597983 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.597962    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 13 21:34:25.598104 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.598002    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-d8kdg\""
Jul 13 21:34:25.598232 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.598212    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 13 21:34:25.598304 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.598246    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 13 21:34:25.598304 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.598264    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 13 21:34:25.598304 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.598275    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 13 21:34:25.598443 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.598370    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8ip4d8dgt08lh\""
Jul 13 21:34:25.609884 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.609864    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"]
Jul 13 21:34:25.645483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.645449    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:25.703221 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.703192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.703321 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.703227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbktc\" (UniqueName: \"kubernetes.io/projected/a4b29830-01ac-438b-86fe-0291e9cd1464-kube-api-access-nbktc\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.703321 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.703249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.703394 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.703329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.703394 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.703355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-tls\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.703455 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.703407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4b29830-01ac-438b-86fe-0291e9cd1464-metrics-client-ca\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.703455 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.703444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.703519 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.703492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-grpc-tls\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.804492 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.804466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.804606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.804515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-grpc-tls\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.804606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.804538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.804606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.804559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbktc\" (UniqueName: \"kubernetes.io/projected/a4b29830-01ac-438b-86fe-0291e9cd1464-kube-api-access-nbktc\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.804761 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.804681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.804815 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.804771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.804878 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.804819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-tls\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.804878 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.804856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4b29830-01ac-438b-86fe-0291e9cd1464-metrics-client-ca\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.805643 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.805616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4b29830-01ac-438b-86fe-0291e9cd1464-metrics-client-ca\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.807330 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.807304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.807474 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.807452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.807601 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.807588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.807637 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.807586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-grpc-tls\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.807745 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.807723    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.807778 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.807765    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a4b29830-01ac-438b-86fe-0291e9cd1464-secret-thanos-querier-tls\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.829656 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.829602    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbktc\" (UniqueName: \"kubernetes.io/projected/a4b29830-01ac-438b-86fe-0291e9cd1464-kube-api-access-nbktc\") pod \"thanos-querier-6997d46fcf-gjb5g\" (UID: \"a4b29830-01ac-438b-86fe-0291e9cd1464\") " pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:25.904578 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:25.904539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:26.032206 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.032176    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"]
Jul 13 21:34:26.036496 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:34:26.036456    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b29830_01ac_438b_86fe_0291e9cd1464.slice/crio-085b71863a6bf56bb654d43509fb961ea1b663fb5fc16d9616113433db901c13 WatchSource:0}: Error finding container 085b71863a6bf56bb654d43509fb961ea1b663fb5fc16d9616113433db901c13: Status 404 returned error can't find the container with id 085b71863a6bf56bb654d43509fb961ea1b663fb5fc16d9616113433db901c13
Jul 13 21:34:26.120081 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.120016    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:26.220753 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.220725    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:26.281162 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.281132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xx6xg" event={"ID":"c2fd3764-1986-4104-98ec-58a05d107526","Type":"ContainerStarted","Data":"faaa1447b74120a76553ae6260336646eb95e0546a2be2b7b6a9fd4fbaa4672c"}
Jul 13 21:34:26.281284 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.281216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xx6xg" event={"ID":"c2fd3764-1986-4104-98ec-58a05d107526","Type":"ContainerStarted","Data":"09c23c465b72686509071d9ee989559e10efd83e56f12d0ae8baf2e1ecee96ac"}
Jul 13 21:34:26.282342 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.282321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g" event={"ID":"a4b29830-01ac-438b-86fe-0291e9cd1464","Type":"ContainerStarted","Data":"085b71863a6bf56bb654d43509fb961ea1b663fb5fc16d9616113433db901c13"}
Jul 13 21:34:26.301417 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.301371    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xx6xg" podStartSLOduration=2.2274490829999998 podStartE2EDuration="3.301357054s" totalImagesPullingTime="1.006620156s" totalInitContainerRuntime="67.287815ms" isStatefulPod=true podCreationTimestamp="2026-07-13 21:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:26.300145748 +0000 UTC m=+307.456539590" watchObservedRunningTime="2026-07-13 21:34:26.301357054 +0000 UTC m=+307.457750928"
Jul 13 21:34:26.549105 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.549057    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:26.648793 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.648743    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:26.864209 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.864121    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:26.961780 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:26.961739    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:27.353684 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:27.353648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:27.357834 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:27.357807    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:34:27.455489 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:27.455447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:27.607716 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:27.607644    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:34:27.656615 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:27.656578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:27.756975 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:27.756943    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:27.857455 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:27.857424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:28.009930 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.009898    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78b84b988f-t7xp6"]
Jul 13 21:34:28.012395 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.012377    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.015863 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.015843    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3315lqk612abu\""
Jul 13 21:34:28.015952 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.015859    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xmjw9\""
Jul 13 21:34:28.016287 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.016129    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 13 21:34:28.016287 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.016143    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 13 21:34:28.016287 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.016188    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 13 21:34:28.016287 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.016234    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 13 21:34:28.024624 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.024603    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78b84b988f-t7xp6"]
Jul 13 21:34:28.127537 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.127511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-audit-log\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.127659 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.127547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-secret-metrics-server-tls\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.127659 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.127569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4jbp\" (UniqueName: \"kubernetes.io/projected/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-kube-api-access-z4jbp\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.127659 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.127588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.127659 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.127649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-client-ca-bundle\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.127873 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.127715    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-secret-metrics-server-client-certs\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.127873 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.127755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-metrics-server-audit-profiles\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.228514 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.228482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-metrics-server-audit-profiles\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.228685 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.228558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-audit-log\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.228685 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.228591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-secret-metrics-server-tls\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.228685 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.228620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4jbp\" (UniqueName: \"kubernetes.io/projected/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-kube-api-access-z4jbp\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.228685 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.228652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.228877 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.228817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-client-ca-bundle\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.228921 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.228875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-secret-metrics-server-client-certs\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.229065 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.229042    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-audit-log\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.229376 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.229343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.229596 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.229577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-metrics-server-audit-profiles\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.231226 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.231201    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-secret-metrics-server-tls\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.231302 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.231274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-client-ca-bundle\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.231379 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.231363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-secret-metrics-server-client-certs\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.236397 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.236380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4jbp\" (UniqueName: \"kubernetes.io/projected/0e3bf464-3a62-41b3-9f4d-ced0b4f179ef-kube-api-access-z4jbp\") pod \"metrics-server-78b84b988f-t7xp6\" (UID: \"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef\") " pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.291967 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.291877    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g" event={"ID":"a4b29830-01ac-438b-86fe-0291e9cd1464","Type":"ContainerStarted","Data":"98d17dba55e5b3ef11306cae6043a00a8e8b89d006d62834cf32055684346a45"}
Jul 13 21:34:28.292110 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.291984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g" event={"ID":"a4b29830-01ac-438b-86fe-0291e9cd1464","Type":"ContainerStarted","Data":"7c0ccefbac035298d3ebe50ab2b964a2506e96e7f96b5a19bc1b06e09c8efaf9"}
Jul 13 21:34:28.292110 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.292023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g" event={"ID":"a4b29830-01ac-438b-86fe-0291e9cd1464","Type":"ContainerStarted","Data":"35eeb18998d4ff8d219fcaa9a088fec01287a18d7e411e72f53bf5049aa99002"}
Jul 13 21:34:28.324172 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.324121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:28.378601 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.378567    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:28.450515 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.450375    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78b84b988f-t7xp6"]
Jul 13 21:34:28.452958 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:34:28.452932    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3bf464_3a62_41b3_9f4d_ced0b4f179ef.slice/crio-174bf9654a57e4841c3a8a486d8eb494f8664ff49755ca4d299c2cbb26ba2bea WatchSource:0}: Error finding container 174bf9654a57e4841c3a8a486d8eb494f8664ff49755ca4d299c2cbb26ba2bea: Status 404 returned error can't find the container with id 174bf9654a57e4841c3a8a486d8eb494f8664ff49755ca4d299c2cbb26ba2bea
Jul 13 21:34:28.453462 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.453431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:28.669062 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:28.668979    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:29.226316 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.226295    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c7b7959d5-wvnrk"
Jul 13 21:34:29.300403 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.300321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g" event={"ID":"a4b29830-01ac-438b-86fe-0291e9cd1464","Type":"ContainerStarted","Data":"b362860b164eba9decabd090b77381baa332c7f204583ebcb187d63c9f0860a5"}
Jul 13 21:34:29.300548 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.300414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g" event={"ID":"a4b29830-01ac-438b-86fe-0291e9cd1464","Type":"ContainerStarted","Data":"0bfb21f134b7b5dbe8c14fad0c9c5ca0afe8e96144620de349f93154e1ab97b4"}
Jul 13 21:34:29.300548 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.300468    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:29.300548 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.300508    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g" event={"ID":"a4b29830-01ac-438b-86fe-0291e9cd1464","Type":"ContainerStarted","Data":"df7e586e32b94cab00cdbc1be7afc4927142032183d2b448524208eadf5bda59"}
Jul 13 21:34:29.302333 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.302306    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6" event={"ID":"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef","Type":"ContainerStarted","Data":"174bf9654a57e4841c3a8a486d8eb494f8664ff49755ca4d299c2cbb26ba2bea"}
Jul 13 21:34:29.340816 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.340770    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g" podStartSLOduration=1.504819923 podStartE2EDuration="4.340752141s" totalImagesPullingTime="2.835932218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:34:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:29.339797308 +0000 UTC m=+310.496191150" watchObservedRunningTime="2026-07-13 21:34:29.340752141 +0000 UTC m=+310.497145983"
Jul 13 21:34:29.384414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.384380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:29.484646 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.484610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:29.805298 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.805262    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:29.811829 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.811803    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 21:34:29.814851 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.814832    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.817503 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.817481    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 13 21:34:29.817620 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.817499    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 13 21:34:29.818394 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.818351    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 13 21:34:29.818818 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.818664    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 13 21:34:29.818900 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.818855    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 13 21:34:29.818900 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.818883    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 13 21:34:29.820413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.819750    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 13 21:34:29.820413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.819750    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 13 21:34:29.820413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.819824    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 13 21:34:29.820413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.819988    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 13 21:34:29.820413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.820144    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9dip9e5nv2h2o\""
Jul 13 21:34:29.820413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.820214    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 13 21:34:29.820413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.820325    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 13 21:34:29.820413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.820416    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xt7vd\""
Jul 13 21:34:29.820845 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.820503    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 13 21:34:29.820951 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.820924    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 13 21:34:29.838438 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.838418    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 21:34:29.944029 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944130 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-config\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944130 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944130 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944130 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-config-out\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944130 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944401 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944401 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944241    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944401 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944401 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944401 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944336    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944401 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944366    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t5ds\" (UniqueName: \"kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-kube-api-access-2t5ds\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:29.944806 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:29.944616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-web-config\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045244 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t5ds\" (UniqueName: \"kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-kube-api-access-2t5ds\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045417 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045417 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045417 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045608 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-web-config\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045684 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045733 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045733 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045705    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-config\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045826 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045826 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045934 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045825    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-config-out\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045934 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.045934 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.046089 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.046089 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.045979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.046089 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.046031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.046089 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.046058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.048208 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.046769    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.048208 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.046663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.048208 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.047316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.048208 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.047347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.048208 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.047382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.048208 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.047437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.048583 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.048317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.048583 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.048391    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.049261 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.049179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.049465 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.049445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-config-out\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.050528 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.049981    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.050528 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.050354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-web-config\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.050528 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.050486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.050748 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.050553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.050805 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.050748    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.050893 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.050865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.051200 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.051139    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.051369 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.051340    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-config\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.051997 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.051974    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.052584 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.052561    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.053121 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.053100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.053255 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.053235    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.053508 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.053493    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t5ds\" (UniqueName: \"kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-kube-api-access-2t5ds\") pod \"prometheus-k8s-0\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.127191 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.127129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:30.258690 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.258531    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 21:34:30.261177 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:34:30.261135    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42bcac94_1498_4dae_81f4_78a720931560.slice/crio-3d82d36235e8e81acc08320100caa995b4a0a537a5948763d66b197ef72a0d57 WatchSource:0}: Error finding container 3d82d36235e8e81acc08320100caa995b4a0a537a5948763d66b197ef72a0d57: Status 404 returned error can't find the container with id 3d82d36235e8e81acc08320100caa995b4a0a537a5948763d66b197ef72a0d57
Jul 13 21:34:30.307242 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.307207    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6" event={"ID":"0e3bf464-3a62-41b3-9f4d-ced0b4f179ef","Type":"ContainerStarted","Data":"ec2d827875e36611557d6cbe3e445c2c72df1e1634147e5dadc4ba029e9dcb89"}
Jul 13 21:34:30.308416 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.308391    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerStarted","Data":"3d82d36235e8e81acc08320100caa995b4a0a537a5948763d66b197ef72a0d57"}
Jul 13 21:34:30.325113 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:30.325070    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6" podStartSLOduration=1.871077994 podStartE2EDuration="3.325056946s" totalImagesPullingTime="1.453978952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:30.323595032 +0000 UTC m=+311.479988870" watchObservedRunningTime="2026-07-13 21:34:30.325056946 +0000 UTC m=+311.481450787"
Jul 13 21:34:31.314127 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:31.314083    2582 generic.go:356] "Generic (PLEG): container finished" podID="42bcac94-1498-4dae-81f4-78a720931560" containerID="a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38" exitCode=0
Jul 13 21:34:31.314535 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:31.314126    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerDied","Data":"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"}
Jul 13 21:34:32.621814 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.621762    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-777b6c774d-p57vr" podUID="915298ab-6dc3-4743-8675-9ae3c60e6699" containerName="registry" containerID="cri-o://fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7" gracePeriod=30
Jul 13 21:34:32.886142 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.886073    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:34:32.975983 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.975945    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmg4d\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-kube-api-access-pmg4d\") pod \"915298ab-6dc3-4743-8675-9ae3c60e6699\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") "
Jul 13 21:34:32.975983 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.975987    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-trusted-ca\") pod \"915298ab-6dc3-4743-8675-9ae3c60e6699\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") "
Jul 13 21:34:32.976245 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.976051    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls\") pod \"915298ab-6dc3-4743-8675-9ae3c60e6699\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") "
Jul 13 21:34:32.976245 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.976185    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-installation-pull-secrets\") pod \"915298ab-6dc3-4743-8675-9ae3c60e6699\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") "
Jul 13 21:34:32.976245 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.976238    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-bound-sa-token\") pod \"915298ab-6dc3-4743-8675-9ae3c60e6699\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") "
Jul 13 21:34:32.976372 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.976307    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/915298ab-6dc3-4743-8675-9ae3c60e6699-ca-trust-extracted\") pod \"915298ab-6dc3-4743-8675-9ae3c60e6699\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") "
Jul 13 21:34:32.976372 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.976336    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-certificates\") pod \"915298ab-6dc3-4743-8675-9ae3c60e6699\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") "
Jul 13 21:34:32.976372 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.976367    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-image-registry-private-configuration\") pod \"915298ab-6dc3-4743-8675-9ae3c60e6699\" (UID: \"915298ab-6dc3-4743-8675-9ae3c60e6699\") "
Jul 13 21:34:32.976844 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.976490    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "915298ab-6dc3-4743-8675-9ae3c60e6699" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:34:32.976844 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.976811    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-trusted-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:34:32.977005 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.976879    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "915298ab-6dc3-4743-8675-9ae3c60e6699" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:34:32.979037 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.978967    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "915298ab-6dc3-4743-8675-9ae3c60e6699" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:34:32.979411 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.979369    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-kube-api-access-pmg4d" (OuterVolumeSpecName: "kube-api-access-pmg4d") pod "915298ab-6dc3-4743-8675-9ae3c60e6699" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699"). InnerVolumeSpecName "kube-api-access-pmg4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:34:32.979411 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.979371    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "915298ab-6dc3-4743-8675-9ae3c60e6699" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:34:32.979411 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.979389    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "915298ab-6dc3-4743-8675-9ae3c60e6699" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:34:32.979411 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.979391    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "915298ab-6dc3-4743-8675-9ae3c60e6699" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:34:32.986932 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:32.986905    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915298ab-6dc3-4743-8675-9ae3c60e6699-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "915298ab-6dc3-4743-8675-9ae3c60e6699" (UID: "915298ab-6dc3-4743-8675-9ae3c60e6699"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:34:33.077826 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.077785    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmg4d\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-kube-api-access-pmg4d\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:34:33.077826 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.077827    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:34:33.078022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.077843    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-installation-pull-secrets\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:34:33.078022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.077859    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/915298ab-6dc3-4743-8675-9ae3c60e6699-bound-sa-token\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:34:33.078022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.077872    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/915298ab-6dc3-4743-8675-9ae3c60e6699-ca-trust-extracted\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:34:33.078022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.077880    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/915298ab-6dc3-4743-8675-9ae3c60e6699-registry-certificates\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:34:33.078022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.077890    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/915298ab-6dc3-4743-8675-9ae3c60e6699-image-registry-private-configuration\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:34:33.322774 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.322733    2582 generic.go:356] "Generic (PLEG): container finished" podID="915298ab-6dc3-4743-8675-9ae3c60e6699" containerID="fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7" exitCode=0
Jul 13 21:34:33.322931 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.322784    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777b6c774d-p57vr"
Jul 13 21:34:33.322931 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.322824    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777b6c774d-p57vr" event={"ID":"915298ab-6dc3-4743-8675-9ae3c60e6699","Type":"ContainerDied","Data":"fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7"}
Jul 13 21:34:33.322931 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.322898    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777b6c774d-p57vr" event={"ID":"915298ab-6dc3-4743-8675-9ae3c60e6699","Type":"ContainerDied","Data":"5772960066fa51e5fe641f7bcbe08d70da41ed22b0f986d8a424f7838e8933ad"}
Jul 13 21:34:33.322931 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.322921    2582 scope.go:117] "RemoveContainer" containerID="fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7"
Jul 13 21:34:33.331066 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.331049    2582 scope.go:117] "RemoveContainer" containerID="fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7"
Jul 13 21:34:33.331354 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:34:33.331334    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7\": container with ID starting with fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7 not found: ID does not exist" containerID="fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7"
Jul 13 21:34:33.331414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.331364    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7"} err="failed to get container status \"fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7\": rpc error: code = NotFound desc = could not find container \"fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7\": container with ID starting with fc506c8bea63ef5d982780843b6c0cf06134e06190670b5516d5a67a631436d7 not found: ID does not exist"
Jul 13 21:34:33.342533 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.342511    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-777b6c774d-p57vr"]
Jul 13 21:34:33.346189 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.346168    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-777b6c774d-p57vr"]
Jul 13 21:34:33.362415 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:33.362391    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="915298ab-6dc3-4743-8675-9ae3c60e6699" path="/var/lib/kubelet/pods/915298ab-6dc3-4743-8675-9ae3c60e6699/volumes"
Jul 13 21:34:35.316186 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:35.316137    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6997d46fcf-gjb5g"
Jul 13 21:34:35.335031 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:35.335004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerStarted","Data":"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"}
Jul 13 21:34:35.335181 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:35.335063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerStarted","Data":"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"}
Jul 13 21:34:35.335181 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:35.335092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerStarted","Data":"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"}
Jul 13 21:34:35.335181 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:35.335119    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerStarted","Data":"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"}
Jul 13 21:34:35.335181 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:35.335142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerStarted","Data":"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"}
Jul 13 21:34:35.335320 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:35.335186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerStarted","Data":"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"}
Jul 13 21:34:35.365423 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:35.365369    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.834609444 podStartE2EDuration="6.365349882s" totalImagesPullingTime="4.509459586s" totalInitContainerRuntime="21.280852ms" isStatefulPod=false podCreationTimestamp="2026-07-13 21:34:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:35.364677869 +0000 UTC m=+316.521071750" watchObservedRunningTime="2026-07-13 21:34:35.365349882 +0000 UTC m=+316.521743727"
Jul 13 21:34:35.546001 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:35.545965    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:35.646877 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:35.646833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:36.280377 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:36.280343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:36.334421 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:36.334390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:37.669524 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:37.669489    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:39.749775 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:39.749739    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:39.848114 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:39.848066    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:40.127935 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:40.127859    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:34:41.469084 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:41.469042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:41.531393 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:41.531339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:43.278505 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:43.278471    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:43.380371 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:43.380336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:45.088585 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:45.088551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:45.185990 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:45.185960    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:46.379580 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:46.379531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:46.480408 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:46.480374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:48.325229 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:48.325188    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:48.325606 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:48.325258    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:34:48.702171 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:48.702063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:48.761258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:48.761224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:48.833227 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:48.833180    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:48.934296 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:48.934263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:50.273605 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:50.273574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:50.374893 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:50.374856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:50.549607 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:50.549513    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:50.650183 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:50.650143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:55.878175 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:55.878125    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:55.932548 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:55.932508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:57.672201 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:57.672099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:34:57.773397 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:34:57.773369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:03.430576 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:03.430545    2582 generic.go:356] "Generic (PLEG): container finished" podID="5fdf28c8-beba-416a-9165-66a689308580" containerID="49ed613ae735e77ac414bb3ba0f160553e5e7fe6e080db6b579a4ce8da9455d4" exitCode=0
Jul 13 21:35:03.430948 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:03.430622    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc" event={"ID":"5fdf28c8-beba-416a-9165-66a689308580","Type":"ContainerDied","Data":"49ed613ae735e77ac414bb3ba0f160553e5e7fe6e080db6b579a4ce8da9455d4"}
Jul 13 21:35:03.431006 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:03.430988    2582 scope.go:117] "RemoveContainer" containerID="49ed613ae735e77ac414bb3ba0f160553e5e7fe6e080db6b579a4ce8da9455d4"
Jul 13 21:35:04.436912 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:04.436877    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-hx5cc" event={"ID":"5fdf28c8-beba-416a-9165-66a689308580","Type":"ContainerStarted","Data":"c67ce9aab2ce4de9dc64854160e45b7e7e0caa5024d5a6b96b7abc818e27b77c"}
Jul 13 21:35:05.546452 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:05.546407    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:05.646581 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:05.646552    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:08.329812 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:08.329781    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:35:08.333633 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:08.333609    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78b84b988f-t7xp6"
Jul 13 21:35:30.127589 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:30.127558    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:30.143006 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:30.142980    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:30.552614 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:30.552586    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:30.879026 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:30.878929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:33.183618 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:33.183581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:35:33.185812 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:33.185794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c50b57-6bb8-4e0f-b2a9-1daab13a74cf-metrics-certs\") pod \"network-metrics-daemon-2njdj\" (UID: \"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf\") " pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:35:33.361775 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:33.361741    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89gzj\""
Jul 13 21:35:33.369945 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:33.369925    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2njdj"
Jul 13 21:35:33.503266 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:33.503213    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2njdj"]
Jul 13 21:35:33.505539 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:35:33.505510    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c50b57_6bb8_4e0f_b2a9_1daab13a74cf.slice/crio-0c5270b0b4219daee818c38c356bef1f4e234f83e07439fdb02c7b336427616e WatchSource:0}: Error finding container 0c5270b0b4219daee818c38c356bef1f4e234f83e07439fdb02c7b336427616e: Status 404 returned error can't find the container with id 0c5270b0b4219daee818c38c356bef1f4e234f83e07439fdb02c7b336427616e
Jul 13 21:35:33.549578 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:33.549545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2njdj" event={"ID":"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf","Type":"ContainerStarted","Data":"0c5270b0b4219daee818c38c356bef1f4e234f83e07439fdb02c7b336427616e"}
Jul 13 21:35:34.554813 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:34.554783    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2njdj" event={"ID":"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf","Type":"ContainerStarted","Data":"6272347100ac693c6a2504dd3c0b23c4e40b72b624d0055d9893d067b8ddb22b"}
Jul 13 21:35:34.903097 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:34.903058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:34.960695 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:34.960661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:35.559508 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:35.559477    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2njdj" event={"ID":"71c50b57-6bb8-4e0f-b2a9-1daab13a74cf","Type":"ContainerStarted","Data":"99cd1170c666cb5575e8c2038644a1af3466cbdf933cc6d0cc8a9256f7cf8b77"}
Jul 13 21:35:35.577291 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:35.577239    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2njdj" podStartSLOduration=375.692491789 podStartE2EDuration="6m16.577209398s" totalImagesPullingTime="884.717609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:35:35.575758756 +0000 UTC m=+376.732152596" watchObservedRunningTime="2026-07-13 21:35:35.577209398 +0000 UTC m=+376.733603242"
Jul 13 21:35:35.655062 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:35.655032    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:35.709057 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:35.709020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:35.766462 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:35.766409    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:39.705610 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:39.705563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:39.769652 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:39.769617    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:47.177544 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:47.177505    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:47.277793 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:47.277760    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:48.209021 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.208985    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 21:35:48.209481 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.209348    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="prometheus" containerID="cri-o://da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20" gracePeriod=600
Jul 13 21:35:48.209481 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.209367    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="thanos-sidecar" containerID="cri-o://b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0" gracePeriod=600
Jul 13 21:35:48.209481 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.209402    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="config-reloader" containerID="cri-o://90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61" gracePeriod=600
Jul 13 21:35:48.209481 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.209420    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy-web" containerID="cri-o://fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137" gracePeriod=600
Jul 13 21:35:48.209481 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.209424    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832" gracePeriod=600
Jul 13 21:35:48.209696 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.209369    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy" containerID="cri-o://2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf" gracePeriod=600
Jul 13 21:35:48.446710 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.446688    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.494274 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494245    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-tls\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.494408 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494290    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-config\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.494408 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494334    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-config-out\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.494408 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494363    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-thanos-prometheus-http-client-file\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.494408 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494407    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-1\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494692    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-db\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494738    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494757    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-metrics-client-ca\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494808    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-kube-rbac-proxy\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494858    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-metrics-client-certs\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494884    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-web-config\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494916    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-2\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494945    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-tls-assets\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.494981    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-trusted-ca-bundle\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.495024    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.495057    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-grpc-tls\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.495081    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.495113    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-serving-certs-ca-bundle\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.495106    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.495167    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-0\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.495193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-kubelet-serving-ca-bundle\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.495218    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t5ds\" (UniqueName: \"kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-kube-api-access-2t5ds\") pod \"42bcac94-1498-4dae-81f4-78a720931560\" (UID: \"42bcac94-1498-4dae-81f4-78a720931560\") "
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.495467    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.495486    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-metrics-client-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.496167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.496725    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.497112    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.497719    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.497852    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.498013    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.498480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-config-out" (OuterVolumeSpecName: "config-out") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:35:48.498866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.498737    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:35:48.499655 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.498858    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:35:48.499655 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.499521    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:35:48.499759 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.499702    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:35:48.500101 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.500068    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:35:48.500354 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.500329    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-config" (OuterVolumeSpecName: "config") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:35:48.500843 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.500798    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:35:48.500934 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.500861    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:35:48.501439 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.501411    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:35:48.501757 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.501731    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-kube-api-access-2t5ds" (OuterVolumeSpecName: "kube-api-access-2t5ds") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "kube-api-access-2t5ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:35:48.510483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.510463    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-web-config" (OuterVolumeSpecName: "web-config") pod "42bcac94-1498-4dae-81f4-78a720931560" (UID: "42bcac94-1498-4dae-81f4-78a720931560"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:35:48.595980 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.595958    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.595980 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.595979    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.595990    2582 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-grpc-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.595999    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596008    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596018    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596026    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596036    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t5ds\" (UniqueName: \"kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-kube-api-access-2t5ds\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596045    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596053    2582 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-config\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596061    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-config-out\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596069    2582 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596077    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-db\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596086    2582 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-kube-rbac-proxy\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596094    2582 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-secret-metrics-client-certs\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596098 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596102    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42bcac94-1498-4dae-81f4-78a720931560-web-config\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596566 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596111    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42bcac94-1498-4dae-81f4-78a720931560-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.596566 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.596119    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42bcac94-1498-4dae-81f4-78a720931560-tls-assets\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:35:48.616970 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.616942    2582 generic.go:356] "Generic (PLEG): container finished" podID="42bcac94-1498-4dae-81f4-78a720931560" containerID="a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832" exitCode=0
Jul 13 21:35:48.616970 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.616966    2582 generic.go:356] "Generic (PLEG): container finished" podID="42bcac94-1498-4dae-81f4-78a720931560" containerID="2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf" exitCode=0
Jul 13 21:35:48.617082 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.616976    2582 generic.go:356] "Generic (PLEG): container finished" podID="42bcac94-1498-4dae-81f4-78a720931560" containerID="fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137" exitCode=0
Jul 13 21:35:48.617082 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.616984    2582 generic.go:356] "Generic (PLEG): container finished" podID="42bcac94-1498-4dae-81f4-78a720931560" containerID="b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0" exitCode=0
Jul 13 21:35:48.617082 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.616991    2582 generic.go:356] "Generic (PLEG): container finished" podID="42bcac94-1498-4dae-81f4-78a720931560" containerID="90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61" exitCode=0
Jul 13 21:35:48.617082 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.617000    2582 generic.go:356] "Generic (PLEG): container finished" podID="42bcac94-1498-4dae-81f4-78a720931560" containerID="da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20" exitCode=0
Jul 13 21:35:48.617082 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.617026    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerDied","Data":"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"}
Jul 13 21:35:48.617277 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.617101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerDied","Data":"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"}
Jul 13 21:35:48.617277 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.617037    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.617277 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.617129    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerDied","Data":"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"}
Jul 13 21:35:48.617277 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.617175    2582 scope.go:117] "RemoveContainer" containerID="a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"
Jul 13 21:35:48.617277 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.617183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerDied","Data":"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"}
Jul 13 21:35:48.617277 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.617222    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerDied","Data":"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"}
Jul 13 21:35:48.617277 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.617248    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerDied","Data":"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"}
Jul 13 21:35:48.617277 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.617279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42bcac94-1498-4dae-81f4-78a720931560","Type":"ContainerDied","Data":"3d82d36235e8e81acc08320100caa995b4a0a537a5948763d66b197ef72a0d57"}
Jul 13 21:35:48.625201 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.625177    2582 scope.go:117] "RemoveContainer" containerID="2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"
Jul 13 21:35:48.632014 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.631998    2582 scope.go:117] "RemoveContainer" containerID="fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"
Jul 13 21:35:48.638460 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.638442    2582 scope.go:117] "RemoveContainer" containerID="b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"
Jul 13 21:35:48.641728 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.641709    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 21:35:48.645583 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.645565    2582 scope.go:117] "RemoveContainer" containerID="90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"
Jul 13 21:35:48.645765 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.645747    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 21:35:48.651793 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.651773    2582 scope.go:117] "RemoveContainer" containerID="da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"
Jul 13 21:35:48.658420 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.658405    2582 scope.go:117] "RemoveContainer" containerID="a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"
Jul 13 21:35:48.664602 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.664584    2582 scope.go:117] "RemoveContainer" containerID="a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"
Jul 13 21:35:48.664847 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:35:48.664828    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": container with ID starting with a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832 not found: ID does not exist" containerID="a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"
Jul 13 21:35:48.664917 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.664858    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"} err="failed to get container status \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": rpc error: code = NotFound desc = could not find container \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": container with ID starting with a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832 not found: ID does not exist"
Jul 13 21:35:48.664917 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.664881    2582 scope.go:117] "RemoveContainer" containerID="2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"
Jul 13 21:35:48.665107 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:35:48.665091    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": container with ID starting with 2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf not found: ID does not exist" containerID="2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"
Jul 13 21:35:48.665145 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.665115    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"} err="failed to get container status \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": rpc error: code = NotFound desc = could not find container \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": container with ID starting with 2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf not found: ID does not exist"
Jul 13 21:35:48.665145 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.665131    2582 scope.go:117] "RemoveContainer" containerID="fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"
Jul 13 21:35:48.665420 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:35:48.665403    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": container with ID starting with fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137 not found: ID does not exist" containerID="fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"
Jul 13 21:35:48.665472 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.665424    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"} err="failed to get container status \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": rpc error: code = NotFound desc = could not find container \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": container with ID starting with fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137 not found: ID does not exist"
Jul 13 21:35:48.665472 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.665437    2582 scope.go:117] "RemoveContainer" containerID="b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"
Jul 13 21:35:48.665642 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:35:48.665625    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": container with ID starting with b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0 not found: ID does not exist" containerID="b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"
Jul 13 21:35:48.665697 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.665649    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"} err="failed to get container status \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": rpc error: code = NotFound desc = could not find container \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": container with ID starting with b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0 not found: ID does not exist"
Jul 13 21:35:48.665697 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.665669    2582 scope.go:117] "RemoveContainer" containerID="90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"
Jul 13 21:35:48.665925 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:35:48.665909    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": container with ID starting with 90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61 not found: ID does not exist" containerID="90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"
Jul 13 21:35:48.665963 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.665934    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"} err="failed to get container status \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": rpc error: code = NotFound desc = could not find container \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": container with ID starting with 90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61 not found: ID does not exist"
Jul 13 21:35:48.665963 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.665952    2582 scope.go:117] "RemoveContainer" containerID="da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"
Jul 13 21:35:48.666162 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:35:48.666134    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": container with ID starting with da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20 not found: ID does not exist" containerID="da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"
Jul 13 21:35:48.666248 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.666225    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"} err="failed to get container status \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": rpc error: code = NotFound desc = could not find container \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": container with ID starting with da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20 not found: ID does not exist"
Jul 13 21:35:48.666287 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.666252    2582 scope.go:117] "RemoveContainer" containerID="a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"
Jul 13 21:35:48.666500 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:35:48.666482    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": container with ID starting with a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38 not found: ID does not exist" containerID="a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"
Jul 13 21:35:48.666569 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.666506    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"} err="failed to get container status \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": rpc error: code = NotFound desc = could not find container \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": container with ID starting with a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38 not found: ID does not exist"
Jul 13 21:35:48.666569 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.666524    2582 scope.go:117] "RemoveContainer" containerID="a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"
Jul 13 21:35:48.666742 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.666724    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"} err="failed to get container status \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": rpc error: code = NotFound desc = could not find container \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": container with ID starting with a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832 not found: ID does not exist"
Jul 13 21:35:48.666789 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.666743    2582 scope.go:117] "RemoveContainer" containerID="2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"
Jul 13 21:35:48.666959 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.666941    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"} err="failed to get container status \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": rpc error: code = NotFound desc = could not find container \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": container with ID starting with 2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf not found: ID does not exist"
Jul 13 21:35:48.667004 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.666959    2582 scope.go:117] "RemoveContainer" containerID="fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"
Jul 13 21:35:48.667199 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.667184    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"} err="failed to get container status \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": rpc error: code = NotFound desc = could not find container \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": container with ID starting with fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137 not found: ID does not exist"
Jul 13 21:35:48.667242 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.667200    2582 scope.go:117] "RemoveContainer" containerID="b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"
Jul 13 21:35:48.667377 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.667361    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"} err="failed to get container status \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": rpc error: code = NotFound desc = could not find container \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": container with ID starting with b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0 not found: ID does not exist"
Jul 13 21:35:48.667416 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.667377    2582 scope.go:117] "RemoveContainer" containerID="90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"
Jul 13 21:35:48.667551 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.667536    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"} err="failed to get container status \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": rpc error: code = NotFound desc = could not find container \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": container with ID starting with 90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61 not found: ID does not exist"
Jul 13 21:35:48.667590 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.667551    2582 scope.go:117] "RemoveContainer" containerID="da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"
Jul 13 21:35:48.667745 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.667727    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"} err="failed to get container status \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": rpc error: code = NotFound desc = could not find container \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": container with ID starting with da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20 not found: ID does not exist"
Jul 13 21:35:48.667784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.667747    2582 scope.go:117] "RemoveContainer" containerID="a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"
Jul 13 21:35:48.667947 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.667932    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"} err="failed to get container status \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": rpc error: code = NotFound desc = could not find container \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": container with ID starting with a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38 not found: ID does not exist"
Jul 13 21:35:48.667988 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.667947    2582 scope.go:117] "RemoveContainer" containerID="a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"
Jul 13 21:35:48.668171 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.668137    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"} err="failed to get container status \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": rpc error: code = NotFound desc = could not find container \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": container with ID starting with a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832 not found: ID does not exist"
Jul 13 21:35:48.668220 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.668171    2582 scope.go:117] "RemoveContainer" containerID="2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"
Jul 13 21:35:48.668403 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.668386    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"} err="failed to get container status \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": rpc error: code = NotFound desc = could not find container \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": container with ID starting with 2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf not found: ID does not exist"
Jul 13 21:35:48.668449 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.668403    2582 scope.go:117] "RemoveContainer" containerID="fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"
Jul 13 21:35:48.668594 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.668577    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"} err="failed to get container status \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": rpc error: code = NotFound desc = could not find container \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": container with ID starting with fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137 not found: ID does not exist"
Jul 13 21:35:48.668628 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.668594    2582 scope.go:117] "RemoveContainer" containerID="b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"
Jul 13 21:35:48.668776 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.668756    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"} err="failed to get container status \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": rpc error: code = NotFound desc = could not find container \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": container with ID starting with b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0 not found: ID does not exist"
Jul 13 21:35:48.668819 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.668776    2582 scope.go:117] "RemoveContainer" containerID="90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"
Jul 13 21:35:48.668965 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.668942    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"} err="failed to get container status \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": rpc error: code = NotFound desc = could not find container \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": container with ID starting with 90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61 not found: ID does not exist"
Jul 13 21:35:48.669002 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.668967    2582 scope.go:117] "RemoveContainer" containerID="da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"
Jul 13 21:35:48.669181 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.669166    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"} err="failed to get container status \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": rpc error: code = NotFound desc = could not find container \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": container with ID starting with da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20 not found: ID does not exist"
Jul 13 21:35:48.669237 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.669181    2582 scope.go:117] "RemoveContainer" containerID="a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"
Jul 13 21:35:48.669391 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.669373    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"} err="failed to get container status \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": rpc error: code = NotFound desc = could not find container \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": container with ID starting with a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38 not found: ID does not exist"
Jul 13 21:35:48.669434 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.669391    2582 scope.go:117] "RemoveContainer" containerID="a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"
Jul 13 21:35:48.669603 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.669585    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"} err="failed to get container status \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": rpc error: code = NotFound desc = could not find container \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": container with ID starting with a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832 not found: ID does not exist"
Jul 13 21:35:48.669669 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.669605    2582 scope.go:117] "RemoveContainer" containerID="2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"
Jul 13 21:35:48.669841 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.669824    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"} err="failed to get container status \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": rpc error: code = NotFound desc = could not find container \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": container with ID starting with 2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf not found: ID does not exist"
Jul 13 21:35:48.669885 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.669841    2582 scope.go:117] "RemoveContainer" containerID="fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"
Jul 13 21:35:48.670035 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.670017    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"} err="failed to get container status \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": rpc error: code = NotFound desc = could not find container \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": container with ID starting with fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137 not found: ID does not exist"
Jul 13 21:35:48.670077 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.670036    2582 scope.go:117] "RemoveContainer" containerID="b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"
Jul 13 21:35:48.670263 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.670245    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"} err="failed to get container status \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": rpc error: code = NotFound desc = could not find container \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": container with ID starting with b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0 not found: ID does not exist"
Jul 13 21:35:48.670318 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.670263    2582 scope.go:117] "RemoveContainer" containerID="90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"
Jul 13 21:35:48.670568 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.670531    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"} err="failed to get container status \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": rpc error: code = NotFound desc = could not find container \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": container with ID starting with 90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61 not found: ID does not exist"
Jul 13 21:35:48.670568 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.670558    2582 scope.go:117] "RemoveContainer" containerID="da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"
Jul 13 21:35:48.670919 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.670889    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"} err="failed to get container status \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": rpc error: code = NotFound desc = could not find container \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": container with ID starting with da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20 not found: ID does not exist"
Jul 13 21:35:48.670919 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.670917    2582 scope.go:117] "RemoveContainer" containerID="a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"
Jul 13 21:35:48.671328 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.671299    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"} err="failed to get container status \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": rpc error: code = NotFound desc = could not find container \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": container with ID starting with a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38 not found: ID does not exist"
Jul 13 21:35:48.671328 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.671323    2582 scope.go:117] "RemoveContainer" containerID="a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"
Jul 13 21:35:48.671581 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.671556    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"} err="failed to get container status \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": rpc error: code = NotFound desc = could not find container \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": container with ID starting with a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832 not found: ID does not exist"
Jul 13 21:35:48.671627 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.671583    2582 scope.go:117] "RemoveContainer" containerID="2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"
Jul 13 21:35:48.671847 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.671809    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"} err="failed to get container status \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": rpc error: code = NotFound desc = could not find container \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": container with ID starting with 2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf not found: ID does not exist"
Jul 13 21:35:48.671847 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.671828    2582 scope.go:117] "RemoveContainer" containerID="fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"
Jul 13 21:35:48.672063 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672039    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"} err="failed to get container status \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": rpc error: code = NotFound desc = could not find container \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": container with ID starting with fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137 not found: ID does not exist"
Jul 13 21:35:48.672063 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672060    2582 scope.go:117] "RemoveContainer" containerID="b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"
Jul 13 21:35:48.672368 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672334    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"} err="failed to get container status \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": rpc error: code = NotFound desc = could not find container \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": container with ID starting with b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0 not found: ID does not exist"
Jul 13 21:35:48.672368 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672353    2582 scope.go:117] "RemoveContainer" containerID="90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"
Jul 13 21:35:48.672469 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672384    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 21:35:48.672553 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672538    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"} err="failed to get container status \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": rpc error: code = NotFound desc = could not find container \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": container with ID starting with 90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61 not found: ID does not exist"
Jul 13 21:35:48.672595 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672552    2582 scope.go:117] "RemoveContainer" containerID="da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"
Jul 13 21:35:48.672652 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672631    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="config-reloader"
Jul 13 21:35:48.672652 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672651    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="config-reloader"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672670    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="init-config-reloader"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672679    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="init-config-reloader"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672688    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="thanos-sidecar"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672697    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="thanos-sidecar"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672709    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy-thanos"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672718    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy-thanos"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672730    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy-web"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672736    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy-web"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672743    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915298ab-6dc3-4743-8675-9ae3c60e6699" containerName="registry"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672748    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="915298ab-6dc3-4743-8675-9ae3c60e6699" containerName="registry"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672756    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="prometheus"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672762    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="prometheus"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672769    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy"
Jul 13 21:35:48.672784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672774    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672828    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="prometheus"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672837    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="config-reloader"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672767    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"} err="failed to get container status \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": rpc error: code = NotFound desc = could not find container \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": container with ID starting with da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20 not found: ID does not exist"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672856    2582 scope.go:117] "RemoveContainer" containerID="a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672843    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy-web"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672905    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy-thanos"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672915    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="thanos-sidecar"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672921    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="915298ab-6dc3-4743-8675-9ae3c60e6699" containerName="registry"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.672927    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bcac94-1498-4dae-81f4-78a720931560" containerName="kube-rbac-proxy"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.673093    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"} err="failed to get container status \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": rpc error: code = NotFound desc = could not find container \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": container with ID starting with a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38 not found: ID does not exist"
Jul 13 21:35:48.673240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.673114    2582 scope.go:117] "RemoveContainer" containerID="a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"
Jul 13 21:35:48.673576 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.673345    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832"} err="failed to get container status \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": rpc error: code = NotFound desc = could not find container \"a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832\": container with ID starting with a714c615f64da0238e810b465815414db9e0d9910169641e74a58548d8a98832 not found: ID does not exist"
Jul 13 21:35:48.673576 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.673363    2582 scope.go:117] "RemoveContainer" containerID="2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"
Jul 13 21:35:48.673643 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.673584    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf"} err="failed to get container status \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": rpc error: code = NotFound desc = could not find container \"2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf\": container with ID starting with 2837e77d5a43a0b723c078866450474ac3b720031a985c6d1f26274d9eb271cf not found: ID does not exist"
Jul 13 21:35:48.673643 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.673607    2582 scope.go:117] "RemoveContainer" containerID="fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"
Jul 13 21:35:48.673800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.673783    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137"} err="failed to get container status \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": rpc error: code = NotFound desc = could not find container \"fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137\": container with ID starting with fff6db50d220480ee0e09153b8c6de6bea3180f57995a3e791cdc3150798a137 not found: ID does not exist"
Jul 13 21:35:48.673845 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.673800    2582 scope.go:117] "RemoveContainer" containerID="b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"
Jul 13 21:35:48.674009 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.673989    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0"} err="failed to get container status \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": rpc error: code = NotFound desc = could not find container \"b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0\": container with ID starting with b36a72ea47636adc8ffb0f83a2bf315e37971c5ffe42b5488f55b6f1ddb7f3e0 not found: ID does not exist"
Jul 13 21:35:48.674009 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.674008    2582 scope.go:117] "RemoveContainer" containerID="90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"
Jul 13 21:35:48.674247 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.674231    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61"} err="failed to get container status \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": rpc error: code = NotFound desc = could not find container \"90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61\": container with ID starting with 90b578deb781a4a0f1ed75aebb1087444f6f4ce4b08fe5509e75f789d0ee8e61 not found: ID does not exist"
Jul 13 21:35:48.674302 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.674247    2582 scope.go:117] "RemoveContainer" containerID="da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"
Jul 13 21:35:48.674492 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.674475    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20"} err="failed to get container status \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": rpc error: code = NotFound desc = could not find container \"da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20\": container with ID starting with da080c855ea1ea1a4cdcc00293c34aeccc76927bd8dfa351fcf424cf68026d20 not found: ID does not exist"
Jul 13 21:35:48.674529 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.674493    2582 scope.go:117] "RemoveContainer" containerID="a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"
Jul 13 21:35:48.674693 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.674677    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38"} err="failed to get container status \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": rpc error: code = NotFound desc = could not find container \"a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38\": container with ID starting with a4822870505708d7b234ae83d751cf3695b9d99b1b4b1e814af2d41596c23d38 not found: ID does not exist"
Jul 13 21:35:48.678345 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.678331    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.681219 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681143    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 13 21:35:48.681341 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681265    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 13 21:35:48.681341 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681332    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9dip9e5nv2h2o\""
Jul 13 21:35:48.681523 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681509    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 13 21:35:48.681593 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681520    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 13 21:35:48.681593 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681530    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xt7vd\""
Jul 13 21:35:48.681593 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681528    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 13 21:35:48.681818 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681803    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 13 21:35:48.681906 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681803    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 13 21:35:48.681906 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681836    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 13 21:35:48.681906 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.681899    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 13 21:35:48.682202 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.682144    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 13 21:35:48.682202 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.682144    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 13 21:35:48.682524 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.682507    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 13 21:35:48.684382 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.684364    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 13 21:35:48.686710 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.686689    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 13 21:35:48.693193 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.693173    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 21:35:48.697251 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697335 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0c89e99-d717-4011-b22a-ba53cba3597c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697335 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697410 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697366    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697410 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697410 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697566 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697566 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697665 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697665 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697665 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697665 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kpvx\" (UniqueName: \"kubernetes.io/projected/a0c89e99-d717-4011-b22a-ba53cba3597c-kube-api-access-4kpvx\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697803 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0c89e99-d717-4011-b22a-ba53cba3597c-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697803 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697696    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697803 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697715    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-config\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697803 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697803 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697969 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697969 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.697969 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.697885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.798686 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.798664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.798787 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.798691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.798787 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.798709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.798787 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.798724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.798787 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.798749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.798787 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.798775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799027 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.798971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799027 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799129 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kpvx\" (UniqueName: \"kubernetes.io/projected/a0c89e99-d717-4011-b22a-ba53cba3597c-kube-api-access-4kpvx\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799129 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0c89e99-d717-4011-b22a-ba53cba3597c-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799129 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799311 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-config\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799311 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799311 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799311 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799311 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799541 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799541 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799541 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0c89e99-d717-4011-b22a-ba53cba3597c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799541 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799541 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799459    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799776 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.799776 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.799762    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.800423 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.800080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.801035 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.800537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.801135 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.801067    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.802288 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.801348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.802288 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.801758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.802288 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.801979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.802288 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.802249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.802700 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.802675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.803009 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.802991    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.803633 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.803571    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0c89e99-d717-4011-b22a-ba53cba3597c-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.803729 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.803675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0c89e99-d717-4011-b22a-ba53cba3597c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.803791 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.803747    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-config\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.804027 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.804005    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.804100 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.804082    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.804530 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.804509    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0c89e99-d717-4011-b22a-ba53cba3597c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.805007 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.804990    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0c89e99-d717-4011-b22a-ba53cba3597c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.807220 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.807130    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kpvx\" (UniqueName: \"kubernetes.io/projected/a0c89e99-d717-4011-b22a-ba53cba3597c-kube-api-access-4kpvx\") pod \"prometheus-k8s-0\" (UID: \"a0c89e99-d717-4011-b22a-ba53cba3597c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:48.987818 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:48.987784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:49.120330 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:49.120283    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 21:35:49.122408 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:35:49.122382    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c89e99_d717_4011_b22a_ba53cba3597c.slice/crio-881bcc0e6f605c18c389ab9d032bbe9cf87d26f1a90fca5a20892bcbb8bca335 WatchSource:0}: Error finding container 881bcc0e6f605c18c389ab9d032bbe9cf87d26f1a90fca5a20892bcbb8bca335: Status 404 returned error can't find the container with id 881bcc0e6f605c18c389ab9d032bbe9cf87d26f1a90fca5a20892bcbb8bca335
Jul 13 21:35:49.362488 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:49.362424    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42bcac94-1498-4dae-81f4-78a720931560" path="/var/lib/kubelet/pods/42bcac94-1498-4dae-81f4-78a720931560/volumes"
Jul 13 21:35:49.622951 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:49.622887    2582 generic.go:356] "Generic (PLEG): container finished" podID="a0c89e99-d717-4011-b22a-ba53cba3597c" containerID="cf2e47a25446589753b886ec3d6c05e4bef7c76b0be31a57b943abd16e2b5339" exitCode=0
Jul 13 21:35:49.623051 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:49.622968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0c89e99-d717-4011-b22a-ba53cba3597c","Type":"ContainerDied","Data":"cf2e47a25446589753b886ec3d6c05e4bef7c76b0be31a57b943abd16e2b5339"}
Jul 13 21:35:49.623051 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:49.623038    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0c89e99-d717-4011-b22a-ba53cba3597c","Type":"ContainerStarted","Data":"881bcc0e6f605c18c389ab9d032bbe9cf87d26f1a90fca5a20892bcbb8bca335"}
Jul 13 21:35:50.630960 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:50.630927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0c89e99-d717-4011-b22a-ba53cba3597c","Type":"ContainerStarted","Data":"f32bf0d82df10e1cfa819ad3a6be4c98dc2a3e45df821075d96914c0c98fe2a1"}
Jul 13 21:35:50.631413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:50.630994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0c89e99-d717-4011-b22a-ba53cba3597c","Type":"ContainerStarted","Data":"d7ec8a74933da322b4b84c155b4fa41b0f60a76dca35f0c660b60989c940d036"}
Jul 13 21:35:50.631413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:50.631029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0c89e99-d717-4011-b22a-ba53cba3597c","Type":"ContainerStarted","Data":"f41c2f70978929bb6902f6f4cd0dd1b587a11e82863a1a464a72e75edae0ba42"}
Jul 13 21:35:50.631413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:50.631081    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0c89e99-d717-4011-b22a-ba53cba3597c","Type":"ContainerStarted","Data":"3a9d50a6ca9481bd942bc17cb6779777d863ec1c189c9ba7d06beaff9ec232b3"}
Jul 13 21:35:50.631413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:50.631113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0c89e99-d717-4011-b22a-ba53cba3597c","Type":"ContainerStarted","Data":"110864d8522b9a35b332c7ea161a2e47186625022b40895fb1b1b912b7dcab1a"}
Jul 13 21:35:50.631413 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:50.631138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0c89e99-d717-4011-b22a-ba53cba3597c","Type":"ContainerStarted","Data":"8cd1288b397e7c98588b82d0d64e3c6e8d17292f0404d43b3b7589eb9b72f865"}
Jul 13 21:35:50.658903 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:50.658848    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6377067199999997 podStartE2EDuration="2.658829423s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.122703ms" isStatefulPod=false podCreationTimestamp="2026-07-13 21:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:35:50.657308766 +0000 UTC m=+391.813702619" watchObservedRunningTime="2026-07-13 21:35:50.658829423 +0000 UTC m=+391.815223277"
Jul 13 21:35:51.599887 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:51.599830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:51.651971 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:51.651927    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:53.988581 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:53.988547    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:35:55.672419 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:55.672382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:55.730805 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:55.730776    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:58.060352 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:58.060320    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:58.113958 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:58.113920    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:35:58.167236 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:35:58.167187    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:05.557871 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:36:05.557833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:05.661165 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:36:05.661118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:05.716034 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:36:05.716003    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:34.564916 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:36:34.564885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:34.665887 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:36:34.665851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:48.988364 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:36:48.988323    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:36:49.003015 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:36:49.002991    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:36:49.868880 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:36:49.868855    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 21:36:50.396101 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:36:50.396061    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:36:50.493173 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:36:50.493122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:17.011772 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:17.011739    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:17.824057 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:17.824023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:20.808210 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:20.808169    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:20.861550 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:20.861515    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:22.337916 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:22.337877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:22.627334 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:22.627252    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:22.728346 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:22.728295    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:22.833523 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:22.833490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:25.117709 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:25.117673    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:25.331973 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:25.331935    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:27.609106 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:27.609068    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:27.934704 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:27.934611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:32.727844 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:32.727809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:33.016510 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:33.016477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:33.117599 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:33.117562    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:35.921791 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:35.921757    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:35.973842 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:35.973813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:38.764474 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:38.764440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:38.868462 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:38.868425    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:41.040497 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:41.040458    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:57.818026 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:57.817994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:57.923458 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:57.923415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:58.437435 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.437396    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:58.724451 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.724357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:58.830025 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.829994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:58.905921 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.905890    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"]
Jul 13 21:37:58.909124 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.909107    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:58.913011 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.912990    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 13 21:37:58.913609 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.913595    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 13 21:37:58.913809 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.913794    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 13 21:37:58.913932 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.913916    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6rfkx\""
Jul 13 21:37:58.918752 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.918734    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 13 21:37:58.935304 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:58.935273    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"]
Jul 13 21:37:59.014634 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.014606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-tmp\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.014797 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.014675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-manifests\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.014797 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.014730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.014797 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.014759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.014797 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.014777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjdwj\" (UniqueName: \"kubernetes.io/projected/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-kube-api-access-wjdwj\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.113327 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.113299    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:59.115242 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.115225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-tmp\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.115298 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.115280    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-manifests\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.115339 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.115309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.115339 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.115326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.115449 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.115342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjdwj\" (UniqueName: \"kubernetes.io/projected/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-kube-api-access-wjdwj\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.115725 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.115705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-manifests\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.117537 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.117504    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-tmp\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.117886 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.117869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.117995 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.117972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.123399 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.123373    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjdwj\" (UniqueName: \"kubernetes.io/projected/8b9d335d-d26b-4539-b5b0-4e7e14d4e99e-kube-api-access-wjdwj\") pod \"opendatahub-operator-controller-manager-5db8865497-l5p4v\" (UID: \"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e\") " pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.209040 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.209010    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:59.218775 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.218754    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:37:59.309426 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.309373    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:59.347594 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.347567    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"]
Jul 13 21:37:59.350040 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:37:59.350002    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9d335d_d26b_4539_b5b0_4e7e14d4e99e.slice/crio-86c83d597f85d50fe95a4c6e93af21a017ae7674aa2622cc56abc06c7d4b1d85 WatchSource:0}: Error finding container 86c83d597f85d50fe95a4c6e93af21a017ae7674aa2622cc56abc06c7d4b1d85: Status 404 returned error can't find the container with id 86c83d597f85d50fe95a4c6e93af21a017ae7674aa2622cc56abc06c7d4b1d85
Jul 13 21:37:59.412266 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.412239    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:59.510925 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.510901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:37:59.610679 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:37:59.610616    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:00.126429 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:00.126390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v" event={"ID":"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e","Type":"ContainerStarted","Data":"86c83d597f85d50fe95a4c6e93af21a017ae7674aa2622cc56abc06c7d4b1d85"}
Jul 13 21:38:03.140856 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:03.140818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v" event={"ID":"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e","Type":"ContainerStarted","Data":"515e2d367b532467e40a5360e40f8628ae28db81764db1d1c653339616cbd93f"}
Jul 13 21:38:04.146330 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:04.146295    2582 generic.go:356] "Generic (PLEG): container finished" podID="8b9d335d-d26b-4539-b5b0-4e7e14d4e99e" containerID="515e2d367b532467e40a5360e40f8628ae28db81764db1d1c653339616cbd93f" exitCode=0
Jul 13 21:38:04.146728 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:04.146383    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v" event={"ID":"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e","Type":"ContainerDied","Data":"515e2d367b532467e40a5360e40f8628ae28db81764db1d1c653339616cbd93f"}
Jul 13 21:38:05.152001 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:05.151966    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v" event={"ID":"8b9d335d-d26b-4539-b5b0-4e7e14d4e99e","Type":"ContainerStarted","Data":"8b652fa47c4d70211ba4898cc1b6072e40f288049aef8f75bf0b7e6ef9b38e63"}
Jul 13 21:38:05.152431 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:05.152089    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:38:05.173114 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:05.173067    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v" podStartSLOduration=2.65058799 podStartE2EDuration="7.173051291s" totalImagesPullingTime="3.436603882s" totalInitContainerRuntime="1.085859419s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:37:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:38:05.171436069 +0000 UTC m=+526.327829911" watchObservedRunningTime="2026-07-13 21:38:05.173051291 +0000 UTC m=+526.329445208"
Jul 13 21:38:09.509754 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:09.509692    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:09.601598 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:09.601561    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:10.841659 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:10.841631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:16.158888 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:16.158814    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5db8865497-l5p4v"
Jul 13 21:38:18.060846 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:18.060813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:18.190681 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:18.190648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:18.539298 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:18.539265    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:18.638508 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:18.638475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:19.260544 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:19.260518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:19.370227 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:19.370194    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:19.594820 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:19.594737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:19.725845 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:19.725813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:20.292605 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:20.292568    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:20.403207 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:20.403171    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:20.467696 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:20.467656    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:20.705367 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:20.705260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:21.446550 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:21.446511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:25.404595 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:25.404551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:25.505073 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:25.505034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:26.094145 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.094110    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"]
Jul 13 21:38:26.100882 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.100857    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.104794 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.104771    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 13 21:38:26.105059 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.105023    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 13 21:38:26.105191 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.105059    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 13 21:38:26.105191 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.105100    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 13 21:38:26.105191 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.105160    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 13 21:38:26.105393 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.105376    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7htvz\""
Jul 13 21:38:26.107677 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.107655    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"]
Jul 13 21:38:26.252753 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.252711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/06f75308-f438-4a9c-9ab9-a2d050827295-manager-config\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.252753 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.252764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/06f75308-f438-4a9c-9ab9-a2d050827295-metrics-cert\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.252986 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.252848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w75v2\" (UniqueName: \"kubernetes.io/projected/06f75308-f438-4a9c-9ab9-a2d050827295-kube-api-access-w75v2\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.252986 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.252879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06f75308-f438-4a9c-9ab9-a2d050827295-cert\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.353800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.353711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06f75308-f438-4a9c-9ab9-a2d050827295-cert\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.353800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.353762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/06f75308-f438-4a9c-9ab9-a2d050827295-manager-config\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.353800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.353797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/06f75308-f438-4a9c-9ab9-a2d050827295-metrics-cert\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.354070 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.353835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w75v2\" (UniqueName: \"kubernetes.io/projected/06f75308-f438-4a9c-9ab9-a2d050827295-kube-api-access-w75v2\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.354542 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.354517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/06f75308-f438-4a9c-9ab9-a2d050827295-manager-config\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.356312 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.356291    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/06f75308-f438-4a9c-9ab9-a2d050827295-metrics-cert\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.356367 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.356316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06f75308-f438-4a9c-9ab9-a2d050827295-cert\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.368570 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.368543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w75v2\" (UniqueName: \"kubernetes.io/projected/06f75308-f438-4a9c-9ab9-a2d050827295-kube-api-access-w75v2\") pod \"lws-controller-manager-789595b4db-zg56j\" (UID: \"06f75308-f438-4a9c-9ab9-a2d050827295\") " pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.410601 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.410559    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:26.539909 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.539881    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"]
Jul 13 21:38:26.542302 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:38:26.542269    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f75308_f438_4a9c_9ab9_a2d050827295.slice/crio-6664b984f38a3ffc119bd428b2e90081c125276641154c547e89818d28352668 WatchSource:0}: Error finding container 6664b984f38a3ffc119bd428b2e90081c125276641154c547e89818d28352668: Status 404 returned error can't find the container with id 6664b984f38a3ffc119bd428b2e90081c125276641154c547e89818d28352668
Jul 13 21:38:26.962461 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.962425    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"]
Jul 13 21:38:26.965837 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.965819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:26.968497 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.968474    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 13 21:38:26.968632 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.968476    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 13 21:38:26.968632 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.968476    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 13 21:38:26.968632 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.968499    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 13 21:38:26.968807 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.968702    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j9p4l\""
Jul 13 21:38:26.977997 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:26.977971    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"]
Jul 13 21:38:27.060846 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.060813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e214029a-2b87-4505-a38d-90a5fd7bfae7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.060993 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.060861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.060993 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.060898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e214029a-2b87-4505-a38d-90a5fd7bfae7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.060993 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.060962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.061108 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.061016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.061108 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.061055    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmb6v\" (UniqueName: \"kubernetes.io/projected/e214029a-2b87-4505-a38d-90a5fd7bfae7-kube-api-access-qmb6v\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.061108 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.061092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.162448 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.162410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e214029a-2b87-4505-a38d-90a5fd7bfae7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.162617 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.162456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.162617 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.162487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.162617 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.162516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmb6v\" (UniqueName: \"kubernetes.io/projected/e214029a-2b87-4505-a38d-90a5fd7bfae7-kube-api-access-qmb6v\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.162617 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.162549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.162617 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.162601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e214029a-2b87-4505-a38d-90a5fd7bfae7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.162865 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.162823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.163259 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.163220    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.164970 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.164944    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.165067 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.164950    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e214029a-2b87-4505-a38d-90a5fd7bfae7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.165067 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.164985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.165067 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.164954    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e214029a-2b87-4505-a38d-90a5fd7bfae7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.172049 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.172020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmb6v\" (UniqueName: \"kubernetes.io/projected/e214029a-2b87-4505-a38d-90a5fd7bfae7-kube-api-access-qmb6v\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.172145 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.172027    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e214029a-2b87-4505-a38d-90a5fd7bfae7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zvmxq\" (UID: \"e214029a-2b87-4505-a38d-90a5fd7bfae7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.235837 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.235808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j" event={"ID":"06f75308-f438-4a9c-9ab9-a2d050827295","Type":"ContainerStarted","Data":"6664b984f38a3ffc119bd428b2e90081c125276641154c547e89818d28352668"}
Jul 13 21:38:27.274937 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.274906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:27.436701 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:27.436433    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"]
Jul 13 21:38:27.439814 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:38:27.439782    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode214029a_2b87_4505_a38d_90a5fd7bfae7.slice/crio-20c5bad7d312658c84a7b5508f1bbbdc505879d37f3bbf63cb1273825fcc5b73 WatchSource:0}: Error finding container 20c5bad7d312658c84a7b5508f1bbbdc505879d37f3bbf63cb1273825fcc5b73: Status 404 returned error can't find the container with id 20c5bad7d312658c84a7b5508f1bbbdc505879d37f3bbf63cb1273825fcc5b73
Jul 13 21:38:28.242567 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:28.242529    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq" event={"ID":"e214029a-2b87-4505-a38d-90a5fd7bfae7","Type":"ContainerStarted","Data":"20c5bad7d312658c84a7b5508f1bbbdc505879d37f3bbf63cb1273825fcc5b73"}
Jul 13 21:38:29.250499 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:29.250451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j" event={"ID":"06f75308-f438-4a9c-9ab9-a2d050827295","Type":"ContainerStarted","Data":"076f92ae41e1fd58deac812971d93a1d3805c9c47b8b276bdeb39626eadac5a8"}
Jul 13 21:38:29.250928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:29.250662    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:29.284259 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:29.284129    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j" podStartSLOduration=0.800994601 podStartE2EDuration="3.28411014s" totalImagesPullingTime="2.483115539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:38:29.282111646 +0000 UTC m=+550.438505488" watchObservedRunningTime="2026-07-13 21:38:29.28411014 +0000 UTC m=+550.440503981"
Jul 13 21:38:29.729670 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:29.729581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:29.843131 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:29.843084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:29.943932 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:29.943890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:30.084901 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:30.084814    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:30.814698 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:30.814655    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 13 21:38:30.815055 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:30.814720    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 13 21:38:30.986733 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:30.986688    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:31.166480 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:31.166389    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:31.224214 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:31.224173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:31.262426 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:31.262391    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq" event={"ID":"e214029a-2b87-4505-a38d-90a5fd7bfae7","Type":"ContainerStarted","Data":"7d5706a6b0ee13dd4d2de6b310cf42a75d081bfbe4de401f87fabf04537fdc51"}
Jul 13 21:38:31.262621 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:31.262603    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:31.264419 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:31.264395    2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zvmxq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 21:38:31.264531 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:31.264451    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq" podUID="e214029a-2b87-4505-a38d-90a5fd7bfae7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 21:38:31.287946 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:31.287887    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq" podStartSLOduration=1.916334811 podStartE2EDuration="5.287865608s" totalImagesPullingTime="3.371530797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:38:31.285018116 +0000 UTC m=+552.441411958" watchObservedRunningTime="2026-07-13 21:38:31.287865608 +0000 UTC m=+552.444259455"
Jul 13 21:38:31.299049 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:31.299015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:31.375427 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:31.375366    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:32.223693 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:32.223658    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:32.267317 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:32.267283    2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zvmxq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 21:38:32.267487 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:32.267342    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq" podUID="e214029a-2b87-4505-a38d-90a5fd7bfae7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 21:38:32.415763 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:32.415732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:32.855061 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:32.855027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:35.267903 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:35.267864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zvmxq"
Jul 13 21:38:40.258420 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:40.258390    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-789595b4db-zg56j"
Jul 13 21:38:50.407748 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:50.407712    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:38:50.508674 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:38:50.508640    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:19.220344 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:19.220310    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:39:19.220928 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:19.220612    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:39:19.226885 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:19.226865    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:39:19.227010 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:19.226995    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:39:19.670479 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:19.670439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:19.740596 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:19.740564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:19.866942 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:19.866907    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:43.285051 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:43.284946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:44.327945 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:44.327904    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:45.327273 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:45.327234    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:54.119300 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:54.119268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:54.171512 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:54.171482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:54.223986 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:54.223952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:56.744388 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:56.744354    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:39:57.343574 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:39:57.343531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:02.557425 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:02.557389    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:04.624750 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:04.624714    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:04.749939 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:04.749908    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:05.156654 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:05.156607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:07.038435 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:07.038394    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:08.778449 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:08.778409    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:10.390090 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:10.390055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:11.978723 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:11.978693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:12.381375 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.381284    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn"]
Jul 13 21:40:12.387229 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.386372    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn"
Jul 13 21:40:12.390130 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.390107    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-k7psq\""
Jul 13 21:40:12.390761 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.390746    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 13 21:40:12.391045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.391028    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 13 21:40:12.391123 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.391068    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 13 21:40:12.398868 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.398845    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn"]
Jul 13 21:40:12.486926 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.486888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdh4\" (UniqueName: \"kubernetes.io/projected/87ea54b2-4db8-4e84-9118-af4827ed9353-kube-api-access-mzdh4\") pod \"dns-operator-controller-manager-864dd78c4c-xcrkn\" (UID: \"87ea54b2-4db8-4e84-9118-af4827ed9353\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn"
Jul 13 21:40:12.588220 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.588174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdh4\" (UniqueName: \"kubernetes.io/projected/87ea54b2-4db8-4e84-9118-af4827ed9353-kube-api-access-mzdh4\") pod \"dns-operator-controller-manager-864dd78c4c-xcrkn\" (UID: \"87ea54b2-4db8-4e84-9118-af4827ed9353\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn"
Jul 13 21:40:12.596396 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.596369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdh4\" (UniqueName: \"kubernetes.io/projected/87ea54b2-4db8-4e84-9118-af4827ed9353-kube-api-access-mzdh4\") pod \"dns-operator-controller-manager-864dd78c4c-xcrkn\" (UID: \"87ea54b2-4db8-4e84-9118-af4827ed9353\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn"
Jul 13 21:40:12.697572 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.697494    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn"
Jul 13 21:40:12.823102 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.823073    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn"]
Jul 13 21:40:12.825817 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:40:12.825791    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ea54b2_4db8_4e84_9118_af4827ed9353.slice/crio-a0ab9b96d4ec0c2319b192b5973828ad62ade6afe145b6d982713e6c415bf620 WatchSource:0}: Error finding container a0ab9b96d4ec0c2319b192b5973828ad62ade6afe145b6d982713e6c415bf620: Status 404 returned error can't find the container with id a0ab9b96d4ec0c2319b192b5973828ad62ade6afe145b6d982713e6c415bf620
Jul 13 21:40:12.827864 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:12.827844    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:40:13.686863 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:13.686823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn" event={"ID":"87ea54b2-4db8-4e84-9118-af4827ed9353","Type":"ContainerStarted","Data":"a0ab9b96d4ec0c2319b192b5973828ad62ade6afe145b6d982713e6c415bf620"}
Jul 13 21:40:13.951656 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:13.951546    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:14.095512 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:14.095474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:14.351251 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:14.351215    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:14.409471 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:14.409439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:15.697365 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:15.697329    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn" event={"ID":"87ea54b2-4db8-4e84-9118-af4827ed9353","Type":"ContainerStarted","Data":"68b58454993a3d6bfdcbd6def4441828089e3cad180877f09210d1c631e53dec"}
Jul 13 21:40:15.697758 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:15.697437    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn"
Jul 13 21:40:15.716047 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:15.715999    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn" podStartSLOduration=1.5208637029999998 podStartE2EDuration="3.715984546s" totalImagesPullingTime="2.195120843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:40:15.715005297 +0000 UTC m=+656.871399140" watchObservedRunningTime="2026-07-13 21:40:15.715984546 +0000 UTC m=+656.872378386"
Jul 13 21:40:17.951648 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:17.951612    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:18.053132 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:18.053098    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:19.342346 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:19.342309    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:19.813886 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:19.813849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:21.157858 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:21.157815    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:21.211351 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:21.211314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:21.963414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:21.963372    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:22.019312 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:22.019279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:26.705984 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:26.705947    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xcrkn"
Jul 13 21:40:27.596257 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:27.596217    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:27.695638 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:27.695599    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:29.637962 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.637931    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"]
Jul 13 21:40:29.644493 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.644470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:29.647088 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.647062    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v6jfl\""
Jul 13 21:40:29.661053 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.661025    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"]
Jul 13 21:40:29.732395 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.732365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgg4g\" (UniqueName: \"kubernetes.io/projected/a2c6e801-3897-45aa-b6c0-b069fb633e78-kube-api-access-kgg4g\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hk4qw\" (UID: \"a2c6e801-3897-45aa-b6c0-b069fb633e78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:29.732549 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.732413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2c6e801-3897-45aa-b6c0-b069fb633e78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hk4qw\" (UID: \"a2c6e801-3897-45aa-b6c0-b069fb633e78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:29.833755 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.833720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgg4g\" (UniqueName: \"kubernetes.io/projected/a2c6e801-3897-45aa-b6c0-b069fb633e78-kube-api-access-kgg4g\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hk4qw\" (UID: \"a2c6e801-3897-45aa-b6c0-b069fb633e78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:29.833906 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.833774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2c6e801-3897-45aa-b6c0-b069fb633e78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hk4qw\" (UID: \"a2c6e801-3897-45aa-b6c0-b069fb633e78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:29.834195 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.834177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2c6e801-3897-45aa-b6c0-b069fb633e78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hk4qw\" (UID: \"a2c6e801-3897-45aa-b6c0-b069fb633e78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:29.842676 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.842651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgg4g\" (UniqueName: \"kubernetes.io/projected/a2c6e801-3897-45aa-b6c0-b069fb633e78-kube-api-access-kgg4g\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hk4qw\" (UID: \"a2c6e801-3897-45aa-b6c0-b069fb633e78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:29.956901 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:29.956819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:30.088525 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:30.088502    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"]
Jul 13 21:40:30.091017 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:40:30.090975    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c6e801_3897_45aa_b6c0_b069fb633e78.slice/crio-084ef5dd6f2ee67f9ff621d3f236441833d0206f1c572732118fbeda50bff128 WatchSource:0}: Error finding container 084ef5dd6f2ee67f9ff621d3f236441833d0206f1c572732118fbeda50bff128: Status 404 returned error can't find the container with id 084ef5dd6f2ee67f9ff621d3f236441833d0206f1c572732118fbeda50bff128
Jul 13 21:40:30.762959 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:30.762920    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw" event={"ID":"a2c6e801-3897-45aa-b6c0-b069fb633e78","Type":"ContainerStarted","Data":"084ef5dd6f2ee67f9ff621d3f236441833d0206f1c572732118fbeda50bff128"}
Jul 13 21:40:33.333461 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:33.333391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:33.388505 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:33.388462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:33.942168 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:33.942109    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:34.044345 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:34.044312    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:34.782710 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:34.782662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw" event={"ID":"a2c6e801-3897-45aa-b6c0-b069fb633e78","Type":"ContainerStarted","Data":"18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9"}
Jul 13 21:40:34.783100 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:34.783040    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:34.807185 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:34.807112    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw" podStartSLOduration=1.8967202090000002 podStartE2EDuration="5.80709472s" totalImagesPullingTime="3.910374511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:40:34.80613233 +0000 UTC m=+675.962526173" watchObservedRunningTime="2026-07-13 21:40:34.80709472 +0000 UTC m=+675.963488561"
Jul 13 21:40:37.364230 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:37.364191    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:37.465364 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:37.465328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:40.782608 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:40.782571    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:40.883620 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:40.883584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:45.788876 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:45.788846    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:50.133438 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:50.133398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:50.185997 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:50.185975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:40:58.297141 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.297104    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"]
Jul 13 21:40:58.297698 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.297406    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw" podUID="a2c6e801-3897-45aa-b6c0-b069fb633e78" containerName="manager" containerID="cri-o://18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9" gracePeriod=10
Jul 13 21:40:58.541958 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.541934    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:58.691645 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.691549    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgg4g\" (UniqueName: \"kubernetes.io/projected/a2c6e801-3897-45aa-b6c0-b069fb633e78-kube-api-access-kgg4g\") pod \"a2c6e801-3897-45aa-b6c0-b069fb633e78\" (UID: \"a2c6e801-3897-45aa-b6c0-b069fb633e78\") "
Jul 13 21:40:58.691800 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.691761    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2c6e801-3897-45aa-b6c0-b069fb633e78-extensions-socket-volume\") pod \"a2c6e801-3897-45aa-b6c0-b069fb633e78\" (UID: \"a2c6e801-3897-45aa-b6c0-b069fb633e78\") "
Jul 13 21:40:58.692238 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.692212    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c6e801-3897-45aa-b6c0-b069fb633e78-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a2c6e801-3897-45aa-b6c0-b069fb633e78" (UID: "a2c6e801-3897-45aa-b6c0-b069fb633e78"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 21:40:58.693710 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.693681    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c6e801-3897-45aa-b6c0-b069fb633e78-kube-api-access-kgg4g" (OuterVolumeSpecName: "kube-api-access-kgg4g") pod "a2c6e801-3897-45aa-b6c0-b069fb633e78" (UID: "a2c6e801-3897-45aa-b6c0-b069fb633e78"). InnerVolumeSpecName "kube-api-access-kgg4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:40:58.793320 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.793286    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgg4g\" (UniqueName: \"kubernetes.io/projected/a2c6e801-3897-45aa-b6c0-b069fb633e78-kube-api-access-kgg4g\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:40:58.793320 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.793314    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2c6e801-3897-45aa-b6c0-b069fb633e78-extensions-socket-volume\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:40:58.888036 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.887997    2582 generic.go:356] "Generic (PLEG): container finished" podID="a2c6e801-3897-45aa-b6c0-b069fb633e78" containerID="18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9" exitCode=0
Jul 13 21:40:58.888215 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.888049    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"
Jul 13 21:40:58.888215 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.888062    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw" event={"ID":"a2c6e801-3897-45aa-b6c0-b069fb633e78","Type":"ContainerDied","Data":"18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9"}
Jul 13 21:40:58.888215 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.888113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw" event={"ID":"a2c6e801-3897-45aa-b6c0-b069fb633e78","Type":"ContainerDied","Data":"084ef5dd6f2ee67f9ff621d3f236441833d0206f1c572732118fbeda50bff128"}
Jul 13 21:40:58.888215 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.888145    2582 scope.go:117] "RemoveContainer" containerID="18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9"
Jul 13 21:40:58.897117 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.897093    2582 scope.go:117] "RemoveContainer" containerID="18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9"
Jul 13 21:40:58.897457 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:40:58.897435    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9\": container with ID starting with 18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9 not found: ID does not exist" containerID="18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9"
Jul 13 21:40:58.897535 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.897468    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9"} err="failed to get container status \"18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9\": rpc error: code = NotFound desc = could not find container \"18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9\": container with ID starting with 18903ccacbdda8c43452d469eeeab7fa24f1c46526e5bdace7ad5d5900f8e3d9 not found: ID does not exist"
Jul 13 21:40:58.926886 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.923000    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"]
Jul 13 21:40:58.932596 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:58.932568    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hk4qw"]
Jul 13 21:40:59.364202 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:40:59.364137    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c6e801-3897-45aa-b6c0-b069fb633e78" path="/var/lib/kubelet/pods/a2c6e801-3897-45aa-b6c0-b069fb633e78/volumes"
Jul 13 21:41:02.283514 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:02.283476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:02.484414 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:02.484367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:18.842252 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:18.842132    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:18.949503 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:18.949439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:19.050712 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.050680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:19.287616 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.287587    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:41:19.287948 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.287935    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c6e801-3897-45aa-b6c0-b069fb633e78" containerName="manager"
Jul 13 21:41:19.287989 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.287950    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c6e801-3897-45aa-b6c0-b069fb633e78" containerName="manager"
Jul 13 21:41:19.288022 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.288015    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c6e801-3897-45aa-b6c0-b069fb633e78" containerName="manager"
Jul 13 21:41:19.290294 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.290278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-np646"
Jul 13 21:41:19.292788 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.292766    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 13 21:41:19.292923 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.292908    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mw5h6\""
Jul 13 21:41:19.299486 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.299458    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:41:19.314991 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.314970    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:41:19.366475 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.366452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a1b0068-ddc3-4234-83cc-151c97892bdc-config-file\") pod \"limitador-limitador-54d6d988d5-np646\" (UID: \"5a1b0068-ddc3-4234-83cc-151c97892bdc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-np646"
Jul 13 21:41:19.366598 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.366491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scpkp\" (UniqueName: \"kubernetes.io/projected/5a1b0068-ddc3-4234-83cc-151c97892bdc-kube-api-access-scpkp\") pod \"limitador-limitador-54d6d988d5-np646\" (UID: \"5a1b0068-ddc3-4234-83cc-151c97892bdc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-np646"
Jul 13 21:41:19.467650 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.467618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a1b0068-ddc3-4234-83cc-151c97892bdc-config-file\") pod \"limitador-limitador-54d6d988d5-np646\" (UID: \"5a1b0068-ddc3-4234-83cc-151c97892bdc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-np646"
Jul 13 21:41:19.467853 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.467674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpkp\" (UniqueName: \"kubernetes.io/projected/5a1b0068-ddc3-4234-83cc-151c97892bdc-kube-api-access-scpkp\") pod \"limitador-limitador-54d6d988d5-np646\" (UID: \"5a1b0068-ddc3-4234-83cc-151c97892bdc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-np646"
Jul 13 21:41:19.468352 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.468330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a1b0068-ddc3-4234-83cc-151c97892bdc-config-file\") pod \"limitador-limitador-54d6d988d5-np646\" (UID: \"5a1b0068-ddc3-4234-83cc-151c97892bdc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-np646"
Jul 13 21:41:19.475528 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.475508    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scpkp\" (UniqueName: \"kubernetes.io/projected/5a1b0068-ddc3-4234-83cc-151c97892bdc-kube-api-access-scpkp\") pod \"limitador-limitador-54d6d988d5-np646\" (UID: \"5a1b0068-ddc3-4234-83cc-151c97892bdc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-np646"
Jul 13 21:41:19.601586 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.601504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-np646"
Jul 13 21:41:19.697686 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.697652    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:19.728660 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.728634    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:41:19.730616 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:41:19.730588    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1b0068_ddc3_4234_83cc_151c97892bdc.slice/crio-9194b8a1a0ea1cda8d3ba8a38395e8fcc36267bae4fcd9eab774879cf4aa7602 WatchSource:0}: Error finding container 9194b8a1a0ea1cda8d3ba8a38395e8fcc36267bae4fcd9eab774879cf4aa7602: Status 404 returned error can't find the container with id 9194b8a1a0ea1cda8d3ba8a38395e8fcc36267bae4fcd9eab774879cf4aa7602
Jul 13 21:41:19.764618 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.764593    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:19.977318 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:19.977230    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-np646" event={"ID":"5a1b0068-ddc3-4234-83cc-151c97892bdc","Type":"ContainerStarted","Data":"9194b8a1a0ea1cda8d3ba8a38395e8fcc36267bae4fcd9eab774879cf4aa7602"}
Jul 13 21:41:22.995087 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:22.995053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-np646" event={"ID":"5a1b0068-ddc3-4234-83cc-151c97892bdc","Type":"ContainerStarted","Data":"4595d220447fff6e377a43c9b737fa9d26b40511ab12dd9a4ba02df20f18c555"}
Jul 13 21:41:22.995481 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:22.995203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-np646"
Jul 13 21:41:23.009787 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:23.009736    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-np646" podStartSLOduration=1.627738694 podStartE2EDuration="4.00972264s" totalImagesPullingTime="2.381983946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:41:23.009597731 +0000 UTC m=+724.165991574" watchObservedRunningTime="2026-07-13 21:41:23.00972264 +0000 UTC m=+724.166116480"
Jul 13 21:41:31.390208 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:31.390167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:34.000490 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:34.000462    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-np646"
Jul 13 21:41:35.809193 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:35.809142    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:35.904431 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:35.904395    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:36.337143 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:36.337094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:48.687343 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:48.687304    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:49.243764 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:49.243728    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:56.296869 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.296831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:56.316670 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.316626    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b85d5995d-6sm5n"]
Jul 13 21:41:56.327643 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.327617    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b85d5995d-6sm5n"]
Jul 13 21:41:56.327767 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.327705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:41:56.330314 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.330272    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 13 21:41:56.330314 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.330284    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 13 21:41:56.330497 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.330421    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bj87w\""
Jul 13 21:41:56.366386 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.366357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:56.389039 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.389008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2clxk\" (UniqueName: \"kubernetes.io/projected/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-kube-api-access-2clxk\") pod \"maas-controller-6b85d5995d-6sm5n\" (UID: \"30592f7d-2d4f-4d1b-9d60-a4c986e6974f\") " pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:41:56.389178 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.389050    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-webhook-cert\") pod \"maas-controller-6b85d5995d-6sm5n\" (UID: \"30592f7d-2d4f-4d1b-9d60-a4c986e6974f\") " pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:41:56.423558 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.423532    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:56.489418 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.489388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-webhook-cert\") pod \"maas-controller-6b85d5995d-6sm5n\" (UID: \"30592f7d-2d4f-4d1b-9d60-a4c986e6974f\") " pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:41:56.489540 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.489502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2clxk\" (UniqueName: \"kubernetes.io/projected/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-kube-api-access-2clxk\") pod \"maas-controller-6b85d5995d-6sm5n\" (UID: \"30592f7d-2d4f-4d1b-9d60-a4c986e6974f\") " pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:41:56.491866 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.491846    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-webhook-cert\") pod \"maas-controller-6b85d5995d-6sm5n\" (UID: \"30592f7d-2d4f-4d1b-9d60-a4c986e6974f\") " pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:41:56.495201 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.495175    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:56.498038 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.498014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2clxk\" (UniqueName: \"kubernetes.io/projected/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-kube-api-access-2clxk\") pod \"maas-controller-6b85d5995d-6sm5n\" (UID: \"30592f7d-2d4f-4d1b-9d60-a4c986e6974f\") " pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:41:56.638046 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.637965    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:41:56.970337 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:56.970310    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b85d5995d-6sm5n"]
Jul 13 21:41:56.972961 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:41:56.972929    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30592f7d_2d4f_4d1b_9d60_a4c986e6974f.slice/crio-d2e094a9e067cf99348f0b5a552f874792f8a5bcbbce0fdd2b27d7da19dd203e WatchSource:0}: Error finding container d2e094a9e067cf99348f0b5a552f874792f8a5bcbbce0fdd2b27d7da19dd203e: Status 404 returned error can't find the container with id d2e094a9e067cf99348f0b5a552f874792f8a5bcbbce0fdd2b27d7da19dd203e
Jul 13 21:41:57.144560 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:57.144529    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b85d5995d-6sm5n" event={"ID":"30592f7d-2d4f-4d1b-9d60-a4c986e6974f","Type":"ContainerStarted","Data":"d2e094a9e067cf99348f0b5a552f874792f8a5bcbbce0fdd2b27d7da19dd203e"}
Jul 13 21:41:59.292529 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:59.292485    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:41:59.383717 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:41:59.383676    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:00.159833 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:00.159794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b85d5995d-6sm5n" event={"ID":"30592f7d-2d4f-4d1b-9d60-a4c986e6974f","Type":"ContainerStarted","Data":"a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2"}
Jul 13 21:42:00.160021 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:00.159910    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:42:00.176533 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:00.176478    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b85d5995d-6sm5n" podStartSLOduration=1.589327916 podStartE2EDuration="4.176462475s" totalImagesPullingTime="2.587134559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:42:00.175126214 +0000 UTC m=+761.331520056" watchObservedRunningTime="2026-07-13 21:42:00.176462475 +0000 UTC m=+761.332856319"
Jul 13 21:42:10.480171 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:10.480126    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b85d5995d-6sm5n"]
Jul 13 21:42:10.480732 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:10.480484    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b85d5995d-6sm5n" podUID="30592f7d-2d4f-4d1b-9d60-a4c986e6974f" containerName="manager" containerID="cri-o://a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2" gracePeriod=30
Jul 13 21:42:10.482187 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:10.482145    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:42:10.725141 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:10.725113    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:42:10.823623 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:10.823546    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-webhook-cert\") pod \"30592f7d-2d4f-4d1b-9d60-a4c986e6974f\" (UID: \"30592f7d-2d4f-4d1b-9d60-a4c986e6974f\") "
Jul 13 21:42:10.823623 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:10.823617    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2clxk\" (UniqueName: \"kubernetes.io/projected/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-kube-api-access-2clxk\") pod \"30592f7d-2d4f-4d1b-9d60-a4c986e6974f\" (UID: \"30592f7d-2d4f-4d1b-9d60-a4c986e6974f\") "
Jul 13 21:42:10.825709 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:10.825682    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "30592f7d-2d4f-4d1b-9d60-a4c986e6974f" (UID: "30592f7d-2d4f-4d1b-9d60-a4c986e6974f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:42:10.825818 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:10.825726    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-kube-api-access-2clxk" (OuterVolumeSpecName: "kube-api-access-2clxk") pod "30592f7d-2d4f-4d1b-9d60-a4c986e6974f" (UID: "30592f7d-2d4f-4d1b-9d60-a4c986e6974f"). InnerVolumeSpecName "kube-api-access-2clxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:42:10.924737 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:10.924693    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-webhook-cert\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:42:10.924737 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:10.924733    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2clxk\" (UniqueName: \"kubernetes.io/projected/30592f7d-2d4f-4d1b-9d60-a4c986e6974f-kube-api-access-2clxk\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:42:11.217214 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:11.217105    2582 generic.go:356] "Generic (PLEG): container finished" podID="30592f7d-2d4f-4d1b-9d60-a4c986e6974f" containerID="a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2" exitCode=0
Jul 13 21:42:11.217214 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:11.217160    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b85d5995d-6sm5n"
Jul 13 21:42:11.217383 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:11.217196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b85d5995d-6sm5n" event={"ID":"30592f7d-2d4f-4d1b-9d60-a4c986e6974f","Type":"ContainerDied","Data":"a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2"}
Jul 13 21:42:11.217383 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:11.217263    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b85d5995d-6sm5n" event={"ID":"30592f7d-2d4f-4d1b-9d60-a4c986e6974f","Type":"ContainerDied","Data":"d2e094a9e067cf99348f0b5a552f874792f8a5bcbbce0fdd2b27d7da19dd203e"}
Jul 13 21:42:11.217383 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:11.217286    2582 scope.go:117] "RemoveContainer" containerID="a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2"
Jul 13 21:42:11.226397 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:11.226376    2582 scope.go:117] "RemoveContainer" containerID="a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2"
Jul 13 21:42:11.226641 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:42:11.226623    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2\": container with ID starting with a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2 not found: ID does not exist" containerID="a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2"
Jul 13 21:42:11.226683 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:11.226650    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2"} err="failed to get container status \"a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2\": rpc error: code = NotFound desc = could not find container \"a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2\": container with ID starting with a8ac35b50f0dca64199e0460d5c10176d9fd66c7c2f3236ca807c21911a4e6f2 not found: ID does not exist"
Jul 13 21:42:11.238197 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:11.238144    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b85d5995d-6sm5n"]
Jul 13 21:42:11.240872 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:11.240853    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b85d5995d-6sm5n"]
Jul 13 21:42:11.363399 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:11.363368    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30592f7d-2d4f-4d1b-9d60-a4c986e6974f" path="/var/lib/kubelet/pods/30592f7d-2d4f-4d1b-9d60-a4c986e6974f/volumes"
Jul 13 21:42:28.051917 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:28.051881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:30.385686 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:30.385643    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:30.478197 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:30.478160    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:30.786843 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:30.786814    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:30.878652 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:30.878614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:31.287845 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:31.287808    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:31.395600 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:31.395551    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:42:31.423121 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:31.423085    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:40.636871 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:40.636837    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:40.697545 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:40.697515    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:44.137473 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:44.137374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:44.237583 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:44.237548    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:51.372513 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:51.372477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:51.951980 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:51.951943    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:42:58.129594 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.129553    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"]
Jul 13 21:42:58.130083 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.130063    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30592f7d-2d4f-4d1b-9d60-a4c986e6974f" containerName="manager"
Jul 13 21:42:58.130131 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.130084    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="30592f7d-2d4f-4d1b-9d60-a4c986e6974f" containerName="manager"
Jul 13 21:42:58.130201 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.130178    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="30592f7d-2d4f-4d1b-9d60-a4c986e6974f" containerName="manager"
Jul 13 21:42:58.133380 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.133361    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.136718 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.136695    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-b8vwz\""
Jul 13 21:42:58.136718 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.136712    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 13 21:42:58.136911 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.136756    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 13 21:42:58.136911 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.136702    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 13 21:42:58.143403 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.143381    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"]
Jul 13 21:42:58.156834 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.156813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:42:58.252825 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.252794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.252966 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.252839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.252966 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.252953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.253036 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.252982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.253069 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.253043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvdc\" (UniqueName: \"kubernetes.io/projected/476e1bc9-2693-44b7-a940-40db993c4c91-kube-api-access-chvdc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.253069 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.253061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/476e1bc9-2693-44b7-a940-40db993c4c91-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.354399 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.354369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvdc\" (UniqueName: \"kubernetes.io/projected/476e1bc9-2693-44b7-a940-40db993c4c91-kube-api-access-chvdc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.354528 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.354405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/476e1bc9-2693-44b7-a940-40db993c4c91-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.354528 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.354446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.354528 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.354490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.354638 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.354535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.354638 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.354559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.354872 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.354823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.354930 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.354882    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.354930 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.354900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.356887 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.356857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/476e1bc9-2693-44b7-a940-40db993c4c91-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.356887 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.356880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/476e1bc9-2693-44b7-a940-40db993c4c91-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.361812 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.361789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvdc\" (UniqueName: \"kubernetes.io/projected/476e1bc9-2693-44b7-a940-40db993c4c91-kube-api-access-chvdc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtjzj\" (UID: \"476e1bc9-2693-44b7-a940-40db993c4c91\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.444340 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.444262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:42:58.573535 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.573509    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"]
Jul 13 21:42:58.575520 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:42:58.575495    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476e1bc9_2693_44b7_a940_40db993c4c91.slice/crio-b41e41d6879298135fdddc119ae31e44451803f8af8b3e8b36f9c8ece31cd9dc WatchSource:0}: Error finding container b41e41d6879298135fdddc119ae31e44451803f8af8b3e8b36f9c8ece31cd9dc: Status 404 returned error can't find the container with id b41e41d6879298135fdddc119ae31e44451803f8af8b3e8b36f9c8ece31cd9dc
Jul 13 21:42:58.956004 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:58.955965    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:42:59.417986 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:42:59.417945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj" event={"ID":"476e1bc9-2693-44b7-a940-40db993c4c91","Type":"ContainerStarted","Data":"b41e41d6879298135fdddc119ae31e44451803f8af8b3e8b36f9c8ece31cd9dc"}
Jul 13 21:43:00.423694 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:00.423657    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj" event={"ID":"476e1bc9-2693-44b7-a940-40db993c4c91","Type":"ContainerStarted","Data":"5df04cdc419329788e93edaf78561fa334de70de612b5ee2e05c1964cea036f0"}
Jul 13 21:43:00.440073 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:00.440026    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj" podStartSLOduration=1.470563106 podStartE2EDuration="2.440002191s" totalImagesPullingTime="969.439085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:43:00.439037285 +0000 UTC m=+821.595431139" watchObservedRunningTime="2026-07-13 21:43:00.440002191 +0000 UTC m=+821.596396033"
Jul 13 21:43:07.264589 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:07.264556    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:43:08.444622 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:08.444582    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:43:08.445014 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:08.444642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:43:08.457717 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:08.457686    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:43:08.469258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:08.469228    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtjzj"
Jul 13 21:43:08.944927 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:08.944884    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:43:16.762501 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:16.762469    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:43:16.861903 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:16.861858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:43:18.148429 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:18.148393    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:43:27.163367 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:27.163333    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:43:29.148863 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:29.148823    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:43:42.868823 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:42.868774    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:43:45.252530 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:43:45.252489    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:44:19.248649 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:44:19.248570    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:44:19.250262 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:44:19.250239    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:44:19.253498 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:44:19.253475    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:44:19.255015 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:44:19.254989    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:44:27.147209 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:44:27.147164    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:27.247274 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:44:27.247238    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:28.319092 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:44:28.319053    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:28.426298 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:44:28.426264    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:49.153079 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:44:49.153038    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:44:55.457113 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:44:55.457080    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:45:00.139877 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:00.139839    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w"]
Jul 13 21:45:00.143187 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:00.143166    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w"
Jul 13 21:45:00.145577 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:00.145557    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 21:45:00.151754 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:00.151730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w"]
Jul 13 21:45:00.155438 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:00.155415    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w"
Jul 13 21:45:00.281593 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:00.281567    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w"]
Jul 13 21:45:00.282995 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:45:00.282963    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767c4252_12fa_41a6_a81a_e73d38ab0162.slice/crio-0d9bfdf7ccc24b1df451a5e46e4495931b9efa9c1b959ab0f86def3584f80ec1 WatchSource:0}: Error finding container 0d9bfdf7ccc24b1df451a5e46e4495931b9efa9c1b959ab0f86def3584f80ec1: Status 404 returned error can't find the container with id 0d9bfdf7ccc24b1df451a5e46e4495931b9efa9c1b959ab0f86def3584f80ec1
Jul 13 21:45:00.939077 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:00.939044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w" event={"ID":"767c4252-12fa-41a6-a81a-e73d38ab0162","Type":"ContainerStarted","Data":"0d9bfdf7ccc24b1df451a5e46e4495931b9efa9c1b959ab0f86def3584f80ec1"}
Jul 13 21:45:02.950124 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:02.950086    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w" event={"ID":"767c4252-12fa-41a6-a81a-e73d38ab0162","Type":"ContainerStarted","Data":"0cb52ec1726489f6ee818939e119d5f9e74fbb9bfd63914e65a3888395afb28e"}
Jul 13 21:45:02.966165 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:02.966092    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w" podStartSLOduration=0.775102363 podStartE2EDuration="2.966072319s" totalImagesPullingTime="2.190969956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:45:02.965879223 +0000 UTC m=+944.122273063" watchObservedRunningTime="2026-07-13 21:45:02.966072319 +0000 UTC m=+944.122466158"
Jul 13 21:45:03.055118 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:03.055038    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:03.104492 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:03.104439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:03.956313 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:03.956272    2582 generic.go:356] "Generic (PLEG): container finished" podID="767c4252-12fa-41a6-a81a-e73d38ab0162" containerID="0cb52ec1726489f6ee818939e119d5f9e74fbb9bfd63914e65a3888395afb28e" exitCode=0
Jul 13 21:45:03.956686 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:03.956358    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w" event={"ID":"767c4252-12fa-41a6-a81a-e73d38ab0162","Type":"ContainerDied","Data":"0cb52ec1726489f6ee818939e119d5f9e74fbb9bfd63914e65a3888395afb28e"}
Jul 13 21:45:04.545711 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:04.545680    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:45:05.090293 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:05.090269    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w"
Jul 13 21:45:05.966559 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:05.966528    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w"
Jul 13 21:45:05.966767 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:05.966524    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732985-j4z9w" event={"ID":"767c4252-12fa-41a6-a81a-e73d38ab0162","Type":"ContainerDied","Data":"0d9bfdf7ccc24b1df451a5e46e4495931b9efa9c1b959ab0f86def3584f80ec1"}
Jul 13 21:45:05.966767 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:05.966678    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d9bfdf7ccc24b1df451a5e46e4495931b9efa9c1b959ab0f86def3584f80ec1"
Jul 13 21:45:16.344180 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:16.344132    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:45:24.955635 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:24.955599    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:45:35.844567 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:35.844524    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:45:38.187512 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:38.187476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:39.575422 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.575380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:45:39.709767 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.709728    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79f5876b6c-dl5fc"]
Jul 13 21:45:39.710116 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.710100    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="767c4252-12fa-41a6-a81a-e73d38ab0162" containerName="cleanup"
Jul 13 21:45:39.710116 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.710118    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="767c4252-12fa-41a6-a81a-e73d38ab0162" containerName="cleanup"
Jul 13 21:45:39.710253 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.710213    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="767c4252-12fa-41a6-a81a-e73d38ab0162" containerName="cleanup"
Jul 13 21:45:39.717682 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.717659    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f5876b6c-dl5fc"
Jul 13 21:45:39.721863 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.721832    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 13 21:45:39.721999 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.721899    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bj87w\""
Jul 13 21:45:39.722168 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.722134    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 13 21:45:39.722902 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.722877    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f5876b6c-dl5fc"]
Jul 13 21:45:39.883110 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.883029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e49c6573-43e7-4736-b00e-c4cc48492b59-webhook-cert\") pod \"maas-controller-79f5876b6c-dl5fc\" (UID: \"e49c6573-43e7-4736-b00e-c4cc48492b59\") " pod="opendatahub/maas-controller-79f5876b6c-dl5fc"
Jul 13 21:45:39.883110 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.883067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/e49c6573-43e7-4736-b00e-c4cc48492b59-kube-api-access-99lgf\") pod \"maas-controller-79f5876b6c-dl5fc\" (UID: \"e49c6573-43e7-4736-b00e-c4cc48492b59\") " pod="opendatahub/maas-controller-79f5876b6c-dl5fc"
Jul 13 21:45:39.983607 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.983566    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e49c6573-43e7-4736-b00e-c4cc48492b59-webhook-cert\") pod \"maas-controller-79f5876b6c-dl5fc\" (UID: \"e49c6573-43e7-4736-b00e-c4cc48492b59\") " pod="opendatahub/maas-controller-79f5876b6c-dl5fc"
Jul 13 21:45:39.983607 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.983611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/e49c6573-43e7-4736-b00e-c4cc48492b59-kube-api-access-99lgf\") pod \"maas-controller-79f5876b6c-dl5fc\" (UID: \"e49c6573-43e7-4736-b00e-c4cc48492b59\") " pod="opendatahub/maas-controller-79f5876b6c-dl5fc"
Jul 13 21:45:39.986045 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.986015    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e49c6573-43e7-4736-b00e-c4cc48492b59-webhook-cert\") pod \"maas-controller-79f5876b6c-dl5fc\" (UID: \"e49c6573-43e7-4736-b00e-c4cc48492b59\") " pod="opendatahub/maas-controller-79f5876b6c-dl5fc"
Jul 13 21:45:39.992165 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:39.992128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99lgf\" (UniqueName: \"kubernetes.io/projected/e49c6573-43e7-4736-b00e-c4cc48492b59-kube-api-access-99lgf\") pod \"maas-controller-79f5876b6c-dl5fc\" (UID: \"e49c6573-43e7-4736-b00e-c4cc48492b59\") " pod="opendatahub/maas-controller-79f5876b6c-dl5fc"
Jul 13 21:45:40.031070 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:40.031040    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f5876b6c-dl5fc"
Jul 13 21:45:40.364819 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:40.364796    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f5876b6c-dl5fc"]
Jul 13 21:45:40.366972 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:45:40.366945    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49c6573_43e7_4736_b00e_c4cc48492b59.slice/crio-ec10256fc89848d467debace76ed7ef39c7ced37b7a30490506de18b63743b8b WatchSource:0}: Error finding container ec10256fc89848d467debace76ed7ef39c7ced37b7a30490506de18b63743b8b: Status 404 returned error can't find the container with id ec10256fc89848d467debace76ed7ef39c7ced37b7a30490506de18b63743b8b
Jul 13 21:45:40.368635 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:40.368618    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:45:41.126309 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:41.126275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f5876b6c-dl5fc" event={"ID":"e49c6573-43e7-4736-b00e-c4cc48492b59","Type":"ContainerStarted","Data":"261d79bc3fa4e531d64bcae2a5ebc9bafceac76abec8aa58340a79788c338475"}
Jul 13 21:45:41.126693 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:41.126324    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f5876b6c-dl5fc" event={"ID":"e49c6573-43e7-4736-b00e-c4cc48492b59","Type":"ContainerStarted","Data":"ec10256fc89848d467debace76ed7ef39c7ced37b7a30490506de18b63743b8b"}
Jul 13 21:45:41.126693 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:41.126410    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79f5876b6c-dl5fc"
Jul 13 21:45:41.143472 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:41.143425    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79f5876b6c-dl5fc" podStartSLOduration=1.606395864 podStartE2EDuration="2.14340966s" totalImagesPullingTime="537.013796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:45:41.143020492 +0000 UTC m=+982.299414333" watchObservedRunningTime="2026-07-13 21:45:41.14340966 +0000 UTC m=+982.299803500"
Jul 13 21:45:52.132476 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:45:52.132443    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79f5876b6c-dl5fc"
Jul 13 21:46:15.503920 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:15.503886    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:15.608037 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:15.608009    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:15.709124 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:15.709090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:21.315593 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:21.315558    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:21.454590 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:21.454554    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:21.546298 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:21.546264    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:21.649467 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:21.649398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:27.338145 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:27.338109    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:27.439421 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:27.439383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:29.717982 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:29.717946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:29.772386 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:29.772354    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:46:44.140262 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:44.140228    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:46:53.845081 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:46:53.845046    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:47:04.638614 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:47:04.638572    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:47:13.144563 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:47:13.144475    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:47:22.589258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:47:22.589218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:22.689730 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:47:22.689672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:47:24.339227 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:47:24.339191    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:47:33.745616 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:47:33.745578    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:47:44.839034 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:47:44.838993    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:47:54.650172 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:47:54.650110    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:48:20.542748 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:48:20.542706    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:48:32.344799 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:48:32.344760    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:48:55.842626 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:48:55.842531    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:49:06.048748 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:49:06.048710    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:49:19.274408 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:49:19.274372    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:49:19.277403 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:49:19.277378    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:49:19.279102 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:49:19.279083    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:49:19.281946 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:49:19.281927    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:49:21.154483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:49:21.154440    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:49:31.957380 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:49:31.957347    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:49:41.654537 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:49:41.654504    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:49:51.041951 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:49:51.041914    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:50:01.346917 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:50:01.346876    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:51:45.103504 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:51:45.103423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:51:48.554488 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:51:48.554460    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:52:00.260998 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:52:00.260965    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:52:09.448293 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:52:09.448255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:52:20.151590 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:52:20.151551    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:52:32.139835 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:52:32.139801    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:52:42.547354 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:52:42.547317    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:52:51.759114 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:52:51.759077    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:53:02.447906 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:53:02.447863    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:53:13.346949 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:53:13.346863    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:53:22.245634 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:53:22.245598    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:53:32.853936 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:53:32.853902    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:53:57.147797 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:53:57.147763    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:54:02.372039 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:02.372005    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:54:04.958064 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:04.958025    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:54:15.340871 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:15.340838    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:54:19.301526 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:19.301501    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:54:19.304104 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:19.304085    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:54:19.306457 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:19.306435    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:54:19.308992 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:19.308976    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:54:24.648577 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:24.648536    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:54:34.451048 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:34.451015    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:54:44.144483 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:44.144442    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:54:55.251844 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:54:55.251808    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:55:06.343836 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:55:06.343798    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:55:14.359454 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:55:14.359416    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:55:23.943036 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:55:23.942998    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:55:33.945315 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:55:33.945280    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:55:48.246569 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:55:48.246536    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:55:56.849697 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:55:56.849657    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:56:06.250240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:56:06.250201    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:56:15.650884 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:56:15.650799    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:56:27.758833 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:56:27.758798    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:56:38.656710 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:56:38.656668    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:56:50.552198 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:56:50.552161    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:57:02.149381 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:02.149342    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:57:13.363189 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:13.363138    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:57:16.274989 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:16.274957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:21.535506 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:21.535456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:25.815620 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:25.815587    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:26.752431 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:26.752394    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:57:34.097781 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:34.097744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:39.210674 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:39.210635    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:42.305645 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:42.305570    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:42.447589 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:42.447555    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:42.778248 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:42.778213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:45.257240 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:45.257202    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:57:48.829763 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:48.829723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:48.911888 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:48.911849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:49.073877 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:49.073828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:49.169023 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:49.168946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:50.245283 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.245246    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:50.345834 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.345796    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:50.385885 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.385847    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"]
Jul 13 21:57:50.389114 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.389092    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:50.392540 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.392516    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 21:57:50.392671 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.392585    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-56bf06f8-serving-cert\""
Jul 13 21:57:50.392793 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.392780    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kfxsg\""
Jul 13 21:57:50.393025 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.393009    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 21:57:50.399965 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.399935    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"]
Jul 13 21:57:50.480549 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.480517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:50.507691 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.507627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f298ef2-12b2-4a58-939d-fb08915392d1-openshift-service-ca\") pod \"maas-api-e2e-ait-56bf06f8-746488747-vrnf6\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:50.507691 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.507659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xkh7\" (UniqueName: \"kubernetes.io/projected/6f298ef2-12b2-4a58-939d-fb08915392d1-kube-api-access-9xkh7\") pod \"maas-api-e2e-ait-56bf06f8-746488747-vrnf6\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:50.507691 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.507692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f298ef2-12b2-4a58-939d-fb08915392d1-maas-api-tls\") pod \"maas-api-e2e-ait-56bf06f8-746488747-vrnf6\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:50.608420 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.608392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f298ef2-12b2-4a58-939d-fb08915392d1-openshift-service-ca\") pod \"maas-api-e2e-ait-56bf06f8-746488747-vrnf6\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:50.608555 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.608423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xkh7\" (UniqueName: \"kubernetes.io/projected/6f298ef2-12b2-4a58-939d-fb08915392d1-kube-api-access-9xkh7\") pod \"maas-api-e2e-ait-56bf06f8-746488747-vrnf6\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:50.608555 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.608446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f298ef2-12b2-4a58-939d-fb08915392d1-maas-api-tls\") pod \"maas-api-e2e-ait-56bf06f8-746488747-vrnf6\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:50.608643 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:57:50.608571    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-serving-cert: secret "maas-api-e2e-ait-56bf06f8-serving-cert" not found
Jul 13 21:57:50.608643 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:57:50.608628    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f298ef2-12b2-4a58-939d-fb08915392d1-maas-api-tls podName:6f298ef2-12b2-4a58-939d-fb08915392d1 nodeName:}" failed. No retries permitted until 2026-07-13 21:57:51.108613661 +0000 UTC m=+1712.265007480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f298ef2-12b2-4a58-939d-fb08915392d1-maas-api-tls") pod "maas-api-e2e-ait-56bf06f8-746488747-vrnf6" (UID: "6f298ef2-12b2-4a58-939d-fb08915392d1") : secret "maas-api-e2e-ait-56bf06f8-serving-cert" not found
Jul 13 21:57:50.609025 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.609006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f298ef2-12b2-4a58-939d-fb08915392d1-openshift-service-ca\") pod \"maas-api-e2e-ait-56bf06f8-746488747-vrnf6\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:50.625730 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.625700    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xkh7\" (UniqueName: \"kubernetes.io/projected/6f298ef2-12b2-4a58-939d-fb08915392d1-kube-api-access-9xkh7\") pod \"maas-api-e2e-ait-56bf06f8-746488747-vrnf6\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:50.757991 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:50.757919    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:51.008916 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:51.008842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:51.107056 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:51.107028    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:51.113028 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:51.113000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f298ef2-12b2-4a58-939d-fb08915392d1-maas-api-tls\") pod \"maas-api-e2e-ait-56bf06f8-746488747-vrnf6\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:51.115356 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:51.115333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f298ef2-12b2-4a58-939d-fb08915392d1-maas-api-tls\") pod \"maas-api-e2e-ait-56bf06f8-746488747-vrnf6\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:51.299364 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:51.299280    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:51.433741 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:51.433653    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"]
Jul 13 21:57:51.436450 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:57:51.436422    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f298ef2_12b2_4a58_939d_fb08915392d1.slice/crio-492beb6614835b0bf2f725f0561138e58a8cb44bb060bc9e45c8b4835d40b9e6 WatchSource:0}: Error finding container 492beb6614835b0bf2f725f0561138e58a8cb44bb060bc9e45c8b4835d40b9e6: Status 404 returned error can't find the container with id 492beb6614835b0bf2f725f0561138e58a8cb44bb060bc9e45c8b4835d40b9e6
Jul 13 21:57:51.437885 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:51.437865    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:57:52.324613 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:52.324569    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6" event={"ID":"6f298ef2-12b2-4a58-939d-fb08915392d1","Type":"ContainerStarted","Data":"492beb6614835b0bf2f725f0561138e58a8cb44bb060bc9e45c8b4835d40b9e6"}
Jul 13 21:57:52.899675 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:52.899646    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:57:53.330613 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:53.330579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6" event={"ID":"6f298ef2-12b2-4a58-939d-fb08915392d1","Type":"ContainerStarted","Data":"ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc"}
Jul 13 21:57:53.331001 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:53.330707    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:53.349486 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:53.349440    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6" podStartSLOduration=1.939343648 podStartE2EDuration="3.349424687s" totalImagesPullingTime="1.410081039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:57:53.34806415 +0000 UTC m=+1714.504457990" watchObservedRunningTime="2026-07-13 21:57:53.349424687 +0000 UTC m=+1714.505818570"
Jul 13 21:57:55.311780 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:55.311745    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"]
Jul 13 21:57:55.315405 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:55.315382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"
Jul 13 21:57:55.319212 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:55.319192    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 21:57:55.342287 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:55.342263    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"]
Jul 13 21:57:55.450171 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:55.450130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150c6290-59d5-4a78-b021-8373e60e82d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d\" (UID: \"150c6290-59d5-4a78-b021-8373e60e82d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"
Jul 13 21:57:55.551050 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:55.551024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150c6290-59d5-4a78-b021-8373e60e82d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d\" (UID: \"150c6290-59d5-4a78-b021-8373e60e82d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"
Jul 13 21:57:55.551626 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:55.551606    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150c6290-59d5-4a78-b021-8373e60e82d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d\" (UID: \"150c6290-59d5-4a78-b021-8373e60e82d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"
Jul 13 21:57:55.625196 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:55.625121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"
Jul 13 21:57:55.753768 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:55.753723    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"]
Jul 13 21:57:55.759229 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:57:55.759195    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150c6290_59d5_4a78_b021_8373e60e82d0.slice/crio-e3f30d3ce3db69b29af7d912b9eeeccd5c7755d9f4c1340daf5d75492ba622ed WatchSource:0}: Error finding container e3f30d3ce3db69b29af7d912b9eeeccd5c7755d9f4c1340daf5d75492ba622ed: Status 404 returned error can't find the container with id e3f30d3ce3db69b29af7d912b9eeeccd5c7755d9f4c1340daf5d75492ba622ed
Jul 13 21:57:56.346122 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:56.346081    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d" event={"ID":"150c6290-59d5-4a78-b021-8373e60e82d0","Type":"ContainerStarted","Data":"325cb773c4541499164c5f99e4093e42f5846fd52ded553276d62fed9445ce7b"}
Jul 13 21:57:56.346560 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:56.346192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d" event={"ID":"150c6290-59d5-4a78-b021-8373e60e82d0","Type":"ContainerStarted","Data":"e3f30d3ce3db69b29af7d912b9eeeccd5c7755d9f4c1340daf5d75492ba622ed"}
Jul 13 21:57:56.364295 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:56.364250    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d" podStartSLOduration=1.3642362160000001 podStartE2EDuration="1.364236216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:57:56.361692131 +0000 UTC m=+1717.518085974" watchObservedRunningTime="2026-07-13 21:57:56.364236216 +0000 UTC m=+1717.520630057"
Jul 13 21:57:56.853712 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:56.853672    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:57:57.351727 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:57.351691    2582 generic.go:356] "Generic (PLEG): container finished" podID="150c6290-59d5-4a78-b021-8373e60e82d0" containerID="325cb773c4541499164c5f99e4093e42f5846fd52ded553276d62fed9445ce7b" exitCode=7
Jul 13 21:57:57.352088 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:57.351776    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d" event={"ID":"150c6290-59d5-4a78-b021-8373e60e82d0","Type":"ContainerDied","Data":"325cb773c4541499164c5f99e4093e42f5846fd52ded553276d62fed9445ce7b"}
Jul 13 21:57:57.352088 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:57.351956    2582 scope.go:117] "RemoveContainer" containerID="325cb773c4541499164c5f99e4093e42f5846fd52ded553276d62fed9445ce7b"
Jul 13 21:57:58.358258 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:58.358184    2582 generic.go:356] "Generic (PLEG): container finished" podID="150c6290-59d5-4a78-b021-8373e60e82d0" containerID="6d1efc308d4141a1337537536f6729f1af2d11ab653d29b2f170af98fb78a417" exitCode=7
Jul 13 21:57:58.358609 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:58.358265    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d" event={"ID":"150c6290-59d5-4a78-b021-8373e60e82d0","Type":"ContainerDied","Data":"6d1efc308d4141a1337537536f6729f1af2d11ab653d29b2f170af98fb78a417"}
Jul 13 21:57:58.358609 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:58.358329    2582 scope.go:117] "RemoveContainer" containerID="325cb773c4541499164c5f99e4093e42f5846fd52ded553276d62fed9445ce7b"
Jul 13 21:57:58.358609 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:58.358555    2582 scope.go:117] "RemoveContainer" containerID="6d1efc308d4141a1337537536f6729f1af2d11ab653d29b2f170af98fb78a417"
Jul 13 21:57:58.358781 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:57:58.358758    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d_odh-ai-gateway-infra(150c6290-59d5-4a78-b021-8373e60e82d0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d" podUID="150c6290-59d5-4a78-b021-8373e60e82d0"
Jul 13 21:57:59.341253 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:59.341171    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:57:59.365784 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:57:59.365754    2582 scope.go:117] "RemoveContainer" containerID="6d1efc308d4141a1337537536f6729f1af2d11ab653d29b2f170af98fb78a417"
Jul 13 21:57:59.366206 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:57:59.365957    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d_odh-ai-gateway-infra(150c6290-59d5-4a78-b021-8373e60e82d0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d" podUID="150c6290-59d5-4a78-b021-8373e60e82d0"
Jul 13 21:58:11.357792 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:11.357759    2582 scope.go:117] "RemoveContainer" containerID="6d1efc308d4141a1337537536f6729f1af2d11ab653d29b2f170af98fb78a417"
Jul 13 21:58:12.424836 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:12.424751    2582 generic.go:356] "Generic (PLEG): container finished" podID="150c6290-59d5-4a78-b021-8373e60e82d0" containerID="474d7fb24904d34cde43045d0091c21534559a077aaa662e6ebcbbb72a48fdb6" exitCode=0
Jul 13 21:58:12.424836 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:12.424824    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d" event={"ID":"150c6290-59d5-4a78-b021-8373e60e82d0","Type":"ContainerDied","Data":"474d7fb24904d34cde43045d0091c21534559a077aaa662e6ebcbbb72a48fdb6"}
Jul 13 21:58:12.425323 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:12.424888    2582 scope.go:117] "RemoveContainer" containerID="6d1efc308d4141a1337537536f6729f1af2d11ab653d29b2f170af98fb78a417"
Jul 13 21:58:13.457206 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:13.457166    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"]
Jul 13 21:58:13.565313 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:13.565289    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"
Jul 13 21:58:13.606704 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:13.606675    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150c6290-59d5-4a78-b021-8373e60e82d0-maas-api-service-ca\") pod \"150c6290-59d5-4a78-b021-8373e60e82d0\" (UID: \"150c6290-59d5-4a78-b021-8373e60e82d0\") "
Jul 13 21:58:13.606993 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:13.606971    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150c6290-59d5-4a78-b021-8373e60e82d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "150c6290-59d5-4a78-b021-8373e60e82d0" (UID: "150c6290-59d5-4a78-b021-8373e60e82d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:58:13.709328 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:13.708114    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150c6290-59d5-4a78-b021-8373e60e82d0-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:58:14.435938 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:14.435902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d" event={"ID":"150c6290-59d5-4a78-b021-8373e60e82d0","Type":"ContainerDied","Data":"e3f30d3ce3db69b29af7d912b9eeeccd5c7755d9f4c1340daf5d75492ba622ed"}
Jul 13 21:58:14.436108 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:14.435950    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3f30d3ce3db69b29af7d912b9eeeccd5c7755d9f4c1340daf5d75492ba622ed"
Jul 13 21:58:14.436108 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:14.435913    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"
Jul 13 21:58:14.467227 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:14.467201    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"]
Jul 13 21:58:14.471508 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:14.471477    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-6bm9d"]
Jul 13 21:58:15.363645 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:15.363612    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="150c6290-59d5-4a78-b021-8373e60e82d0" path="/var/lib/kubelet/pods/150c6290-59d5-4a78-b021-8373e60e82d0/volumes"
Jul 13 21:58:45.327177 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.327103    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc"]
Jul 13 21:58:45.327785 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.327440    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150c6290-59d5-4a78-b021-8373e60e82d0" containerName="revoke-keys"
Jul 13 21:58:45.327785 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.327452    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="150c6290-59d5-4a78-b021-8373e60e82d0" containerName="revoke-keys"
Jul 13 21:58:45.327785 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.327476    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150c6290-59d5-4a78-b021-8373e60e82d0" containerName="revoke-keys"
Jul 13 21:58:45.327785 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.327482    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="150c6290-59d5-4a78-b021-8373e60e82d0" containerName="revoke-keys"
Jul 13 21:58:45.327785 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.327535    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="150c6290-59d5-4a78-b021-8373e60e82d0" containerName="revoke-keys"
Jul 13 21:58:45.327785 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.327543    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="150c6290-59d5-4a78-b021-8373e60e82d0" containerName="revoke-keys"
Jul 13 21:58:45.330773 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.330751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc"
Jul 13 21:58:45.333614 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.333595    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 21:58:45.342508 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.342483    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc"]
Jul 13 21:58:45.471637 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.471606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fea04fa6-3653-4578-b357-9260281d910a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc\" (UID: \"fea04fa6-3653-4578-b357-9260281d910a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc"
Jul 13 21:58:45.572077 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.572046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fea04fa6-3653-4578-b357-9260281d910a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc\" (UID: \"fea04fa6-3653-4578-b357-9260281d910a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc"
Jul 13 21:58:45.572692 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.572670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fea04fa6-3653-4578-b357-9260281d910a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc\" (UID: \"fea04fa6-3653-4578-b357-9260281d910a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc"
Jul 13 21:58:45.640767 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.640715    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc"
Jul 13 21:58:45.976472 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:45.976447    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc"]
Jul 13 21:58:45.978809 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:58:45.978784    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea04fa6_3653_4578_b357_9260281d910a.slice/crio-7cb744a1c7b8d371cbb3d5a32120a91e82313b117a12e1170f9ec07b02e21dd1 WatchSource:0}: Error finding container 7cb744a1c7b8d371cbb3d5a32120a91e82313b117a12e1170f9ec07b02e21dd1: Status 404 returned error can't find the container with id 7cb744a1c7b8d371cbb3d5a32120a91e82313b117a12e1170f9ec07b02e21dd1
Jul 13 21:58:46.586903 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:46.586811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc" event={"ID":"fea04fa6-3653-4578-b357-9260281d910a","Type":"ContainerStarted","Data":"11c48ccb6d53d6a3b2abd7fb94dff7848528ceee863109c8e5a6002ee27a0c6a"}
Jul 13 21:58:46.586903 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:46.586889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc" event={"ID":"fea04fa6-3653-4578-b357-9260281d910a","Type":"ContainerStarted","Data":"7cb744a1c7b8d371cbb3d5a32120a91e82313b117a12e1170f9ec07b02e21dd1"}
Jul 13 21:58:46.610552 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:46.610492    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc" podStartSLOduration=1.610479376 podStartE2EDuration="1.610479376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:46.608707151 +0000 UTC m=+1767.765100983" watchObservedRunningTime="2026-07-13 21:58:46.610479376 +0000 UTC m=+1767.766873267"
Jul 13 21:58:47.593662 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:47.593627    2582 generic.go:356] "Generic (PLEG): container finished" podID="fea04fa6-3653-4578-b357-9260281d910a" containerID="11c48ccb6d53d6a3b2abd7fb94dff7848528ceee863109c8e5a6002ee27a0c6a" exitCode=0
Jul 13 21:58:47.593662 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:47.593666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc" event={"ID":"fea04fa6-3653-4578-b357-9260281d910a","Type":"ContainerDied","Data":"11c48ccb6d53d6a3b2abd7fb94dff7848528ceee863109c8e5a6002ee27a0c6a"}
Jul 13 21:58:48.732597 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:48.732573    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc"
Jul 13 21:58:48.901268 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:48.901175    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fea04fa6-3653-4578-b357-9260281d910a-maas-api-service-ca\") pod \"fea04fa6-3653-4578-b357-9260281d910a\" (UID: \"fea04fa6-3653-4578-b357-9260281d910a\") "
Jul 13 21:58:48.901507 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:48.901482    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea04fa6-3653-4578-b357-9260281d910a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fea04fa6-3653-4578-b357-9260281d910a" (UID: "fea04fa6-3653-4578-b357-9260281d910a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:58:49.002582 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:49.002556    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fea04fa6-3653-4578-b357-9260281d910a-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:58:49.604599 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:49.604574    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc"
Jul 13 21:58:49.604599 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:49.604587    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc" event={"ID":"fea04fa6-3653-4578-b357-9260281d910a","Type":"ContainerDied","Data":"7cb744a1c7b8d371cbb3d5a32120a91e82313b117a12e1170f9ec07b02e21dd1"}
Jul 13 21:58:49.604837 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:49.604631    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cb744a1c7b8d371cbb3d5a32120a91e82313b117a12e1170f9ec07b02e21dd1"
Jul 13 21:58:55.872402 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:55.872348    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:58:55.873975 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:55.873942    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"]
Jul 13 21:58:55.874229 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:55.874204    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6" podUID="6f298ef2-12b2-4a58-939d-fb08915392d1" containerName="maas-api" containerID="cri-o://ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc" gracePeriod=30
Jul 13 21:58:55.986453 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:55.986424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:58:56.132937 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.132882    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:58:56.261935 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.261903    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f298ef2-12b2-4a58-939d-fb08915392d1-maas-api-tls\") pod \"6f298ef2-12b2-4a58-939d-fb08915392d1\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") "
Jul 13 21:58:56.262105 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.261941    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xkh7\" (UniqueName: \"kubernetes.io/projected/6f298ef2-12b2-4a58-939d-fb08915392d1-kube-api-access-9xkh7\") pod \"6f298ef2-12b2-4a58-939d-fb08915392d1\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") "
Jul 13 21:58:56.262105 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.261977    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f298ef2-12b2-4a58-939d-fb08915392d1-openshift-service-ca\") pod \"6f298ef2-12b2-4a58-939d-fb08915392d1\" (UID: \"6f298ef2-12b2-4a58-939d-fb08915392d1\") "
Jul 13 21:58:56.262350 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.262330    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f298ef2-12b2-4a58-939d-fb08915392d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f298ef2-12b2-4a58-939d-fb08915392d1" (UID: "6f298ef2-12b2-4a58-939d-fb08915392d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:58:56.264127 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.264098    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f298ef2-12b2-4a58-939d-fb08915392d1-kube-api-access-9xkh7" (OuterVolumeSpecName: "kube-api-access-9xkh7") pod "6f298ef2-12b2-4a58-939d-fb08915392d1" (UID: "6f298ef2-12b2-4a58-939d-fb08915392d1"). InnerVolumeSpecName "kube-api-access-9xkh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:58:56.264239 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.264184    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f298ef2-12b2-4a58-939d-fb08915392d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f298ef2-12b2-4a58-939d-fb08915392d1" (UID: "6f298ef2-12b2-4a58-939d-fb08915392d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:58:56.362553 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.362522    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f298ef2-12b2-4a58-939d-fb08915392d1-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:58:56.362553 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.362547    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xkh7\" (UniqueName: \"kubernetes.io/projected/6f298ef2-12b2-4a58-939d-fb08915392d1-kube-api-access-9xkh7\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:58:56.362709 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.362576    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f298ef2-12b2-4a58-939d-fb08915392d1-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:58:56.639657 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.639627    2582 generic.go:356] "Generic (PLEG): container finished" podID="6f298ef2-12b2-4a58-939d-fb08915392d1" containerID="ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc" exitCode=0
Jul 13 21:58:56.639806 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.639674    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"
Jul 13 21:58:56.639806 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.639683    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6" event={"ID":"6f298ef2-12b2-4a58-939d-fb08915392d1","Type":"ContainerDied","Data":"ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc"}
Jul 13 21:58:56.639806 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.639723    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6" event={"ID":"6f298ef2-12b2-4a58-939d-fb08915392d1","Type":"ContainerDied","Data":"492beb6614835b0bf2f725f0561138e58a8cb44bb060bc9e45c8b4835d40b9e6"}
Jul 13 21:58:56.639806 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.639746    2582 scope.go:117] "RemoveContainer" containerID="ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc"
Jul 13 21:58:56.650778 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.650758    2582 scope.go:117] "RemoveContainer" containerID="ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc"
Jul 13 21:58:56.651033 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:58:56.651006    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc\": container with ID starting with ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc not found: ID does not exist" containerID="ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc"
Jul 13 21:58:56.651111 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.651039    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc"} err="failed to get container status \"ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc\": rpc error: code = NotFound desc = could not find container \"ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc\": container with ID starting with ebd606ad28b45da4eedc2e3e4e0ba4934db117e8f0484d8959ac4527181671dc not found: ID does not exist"
Jul 13 21:58:56.668234 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.668211    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"]
Jul 13 21:58:56.675863 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:56.675840    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56bf06f8-746488747-vrnf6"]
Jul 13 21:58:57.365469 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:57.365430    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f298ef2-12b2-4a58-939d-fb08915392d1" path="/var/lib/kubelet/pods/6f298ef2-12b2-4a58-939d-fb08915392d1/volumes"
Jul 13 21:58:58.767070 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:58:58.767026    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:59:00.424547 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:00.424511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:07.173124 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:07.173084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:07.305199 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:07.305144    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:10.704582 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:10.704542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:11.175480 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:11.175443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:11.363216 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:11.363125    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:11.597753 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:11.597709    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:12.758873 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:12.758828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:13.026636 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.026560    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"]
Jul 13 21:59:13.026940 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.026926    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f298ef2-12b2-4a58-939d-fb08915392d1" containerName="maas-api"
Jul 13 21:59:13.026983 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.026944    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f298ef2-12b2-4a58-939d-fb08915392d1" containerName="maas-api"
Jul 13 21:59:13.026983 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.026964    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150c6290-59d5-4a78-b021-8373e60e82d0" containerName="revoke-keys"
Jul 13 21:59:13.026983 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.026971    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="150c6290-59d5-4a78-b021-8373e60e82d0" containerName="revoke-keys"
Jul 13 21:59:13.027073 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.026984    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fea04fa6-3653-4578-b357-9260281d910a" containerName="revoke-keys"
Jul 13 21:59:13.027073 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.026989    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fea04fa6-3653-4578-b357-9260281d910a" containerName="revoke-keys"
Jul 13 21:59:13.027073 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.027044    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fea04fa6-3653-4578-b357-9260281d910a" containerName="revoke-keys"
Jul 13 21:59:13.027073 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.027053    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f298ef2-12b2-4a58-939d-fb08915392d1" containerName="maas-api"
Jul 13 21:59:13.027073 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.027062    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="150c6290-59d5-4a78-b021-8373e60e82d0" containerName="revoke-keys"
Jul 13 21:59:13.031246 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.031228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.036938 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.036913    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7e874d59-serving-cert\""
Jul 13 21:59:13.037089 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.037075    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 21:59:13.038039 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.038017    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 21:59:13.038039 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.038032    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kfxsg\""
Jul 13 21:59:13.046711 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.046687    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"]
Jul 13 21:59:13.086317 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.086289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzfqh\" (UniqueName: \"kubernetes.io/projected/79f9b815-69aa-4a4c-9d59-5affc5588fda-kube-api-access-fzfqh\") pod \"maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.086431 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.086390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f9b815-69aa-4a4c-9d59-5affc5588fda-openshift-service-ca\") pod \"maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.086431 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.086425    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f9b815-69aa-4a4c-9d59-5affc5588fda-maas-api-tls\") pod \"maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.116594 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.116566    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:13.186841 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.186815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzfqh\" (UniqueName: \"kubernetes.io/projected/79f9b815-69aa-4a4c-9d59-5affc5588fda-kube-api-access-fzfqh\") pod \"maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.186946 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.186919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f9b815-69aa-4a4c-9d59-5affc5588fda-openshift-service-ca\") pod \"maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.187077 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.186959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f9b815-69aa-4a4c-9d59-5affc5588fda-maas-api-tls\") pod \"maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.187141 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:59:13.187091    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-serving-cert: secret "maas-api-e2e-migrate-7e874d59-serving-cert" not found
Jul 13 21:59:13.187233 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:59:13.187198    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79f9b815-69aa-4a4c-9d59-5affc5588fda-maas-api-tls podName:79f9b815-69aa-4a4c-9d59-5affc5588fda nodeName:}" failed. No retries permitted until 2026-07-13 21:59:13.687175774 +0000 UTC m=+1794.843569607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79f9b815-69aa-4a4c-9d59-5affc5588fda-maas-api-tls") pod "maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh" (UID: "79f9b815-69aa-4a4c-9d59-5affc5588fda") : secret "maas-api-e2e-migrate-7e874d59-serving-cert" not found
Jul 13 21:59:13.187556 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.187536    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f9b815-69aa-4a4c-9d59-5affc5588fda-openshift-service-ca\") pod \"maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.209091 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.209068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzfqh\" (UniqueName: \"kubernetes.io/projected/79f9b815-69aa-4a4c-9d59-5affc5588fda-kube-api-access-fzfqh\") pod \"maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.472856 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.472755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:13.690280 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.690248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f9b815-69aa-4a4c-9d59-5affc5588fda-maas-api-tls\") pod \"maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.692530 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.692503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f9b815-69aa-4a4c-9d59-5affc5588fda-maas-api-tls\") pod \"maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.940566 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.940535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:13.996003 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:13.995405    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:14.096307 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:14.096273    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"]
Jul 13 21:59:14.103116 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:59:14.103089    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f9b815_69aa_4a4c_9d59_5affc5588fda.slice/crio-d51f7468360d5cb78a8f86d3e180b5719c28df394593742249be83c0e815511d WatchSource:0}: Error finding container d51f7468360d5cb78a8f86d3e180b5719c28df394593742249be83c0e815511d: Status 404 returned error can't find the container with id d51f7468360d5cb78a8f86d3e180b5719c28df394593742249be83c0e815511d
Jul 13 21:59:14.142460 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:14.142431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:14.723516 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:14.723479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh" event={"ID":"79f9b815-69aa-4a4c-9d59-5affc5588fda","Type":"ContainerStarted","Data":"3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51"}
Jul 13 21:59:14.723708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:14.723546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh" event={"ID":"79f9b815-69aa-4a4c-9d59-5affc5588fda","Type":"ContainerStarted","Data":"d51f7468360d5cb78a8f86d3e180b5719c28df394593742249be83c0e815511d"}
Jul 13 21:59:14.723708 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:14.723579    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:14.776090 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:14.776041    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh" podStartSLOduration=2.340525659 podStartE2EDuration="2.776022995s" totalImagesPullingTime="435.497336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:14.775922547 +0000 UTC m=+1795.932316388" watchObservedRunningTime="2026-07-13 21:59:14.776022995 +0000 UTC m=+1795.932416838"
Jul 13 21:59:17.154734 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.154698    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"]
Jul 13 21:59:17.158097 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.158078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"
Jul 13 21:59:17.162848 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.162817    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 21:59:17.168370 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.168342    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"]
Jul 13 21:59:17.216359 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.216334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e55ee84-d422-4423-8614-3ebed0c0941d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4\" (UID: \"8e55ee84-d422-4423-8614-3ebed0c0941d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"
Jul 13 21:59:17.245199 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.245171    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:17.316963 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.316925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e55ee84-d422-4423-8614-3ebed0c0941d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4\" (UID: \"8e55ee84-d422-4423-8614-3ebed0c0941d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"
Jul 13 21:59:17.317565 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.317546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e55ee84-d422-4423-8614-3ebed0c0941d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4\" (UID: \"8e55ee84-d422-4423-8614-3ebed0c0941d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"
Jul 13 21:59:17.365244 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.365201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:17.471105 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.471075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"
Jul 13 21:59:17.846088 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:17.846065    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"]
Jul 13 21:59:17.848298 ip-10-0-140-193 kubenswrapper[2582]: W0713 21:59:17.848271    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e55ee84_d422_4423_8614_3ebed0c0941d.slice/crio-f29b9695e92b16353b05f8ce59c7d3d02b08a2ddde9de876f584f99a65b1a73a WatchSource:0}: Error finding container f29b9695e92b16353b05f8ce59c7d3d02b08a2ddde9de876f584f99a65b1a73a: Status 404 returned error can't find the container with id f29b9695e92b16353b05f8ce59c7d3d02b08a2ddde9de876f584f99a65b1a73a
Jul 13 21:59:18.743658 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:18.743620    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerID="43a0fb0bbfe08dcfeb55b15aeb012ee0ff9c0b935941a62016c8985e835ac48b" exitCode=7
Jul 13 21:59:18.744028 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:18.743661    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4" event={"ID":"8e55ee84-d422-4423-8614-3ebed0c0941d","Type":"ContainerDied","Data":"43a0fb0bbfe08dcfeb55b15aeb012ee0ff9c0b935941a62016c8985e835ac48b"}
Jul 13 21:59:18.744028 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:18.743732    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4" event={"ID":"8e55ee84-d422-4423-8614-3ebed0c0941d","Type":"ContainerStarted","Data":"f29b9695e92b16353b05f8ce59c7d3d02b08a2ddde9de876f584f99a65b1a73a"}
Jul 13 21:59:18.744028 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:18.743895    2582 scope.go:117] "RemoveContainer" containerID="43a0fb0bbfe08dcfeb55b15aeb012ee0ff9c0b935941a62016c8985e835ac48b"
Jul 13 21:59:19.329974 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:19.329951    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:59:19.333869 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:19.333850    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 21:59:19.334904 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:19.334886    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:59:19.338218 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:19.338201    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 21:59:19.749892 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:19.749864    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerID="041f90eccd1aadfc31c9e9a19a4e176ba471c1b102e3f4605fb4f000f40efe73" exitCode=7
Jul 13 21:59:19.750328 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:19.749921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4" event={"ID":"8e55ee84-d422-4423-8614-3ebed0c0941d","Type":"ContainerDied","Data":"041f90eccd1aadfc31c9e9a19a4e176ba471c1b102e3f4605fb4f000f40efe73"}
Jul 13 21:59:19.750328 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:19.749982    2582 scope.go:117] "RemoveContainer" containerID="43a0fb0bbfe08dcfeb55b15aeb012ee0ff9c0b935941a62016c8985e835ac48b"
Jul 13 21:59:19.750328 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:19.750141    2582 scope.go:117] "RemoveContainer" containerID="041f90eccd1aadfc31c9e9a19a4e176ba471c1b102e3f4605fb4f000f40efe73"
Jul 13 21:59:19.750501 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:59:19.750361    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4_odh-ai-gateway-infra(8e55ee84-d422-4423-8614-3ebed0c0941d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d"
Jul 13 21:59:20.734264 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:20.734232    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 21:59:20.757663 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:20.757630    2582 scope.go:117] "RemoveContainer" containerID="041f90eccd1aadfc31c9e9a19a4e176ba471c1b102e3f4605fb4f000f40efe73"
Jul 13 21:59:20.758114 ip-10-0-140-193 kubenswrapper[2582]: E0713 21:59:20.757862    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4_odh-ai-gateway-infra(8e55ee84-d422-4423-8614-3ebed0c0941d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d"
Jul 13 21:59:20.778289 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:20.778263    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 21:59:21.064460 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:21.064420    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 21:59:32.358518 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:32.358476    2582 scope.go:117] "RemoveContainer" containerID="041f90eccd1aadfc31c9e9a19a4e176ba471c1b102e3f4605fb4f000f40efe73"
Jul 13 21:59:32.822571 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:32.822544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4" event={"ID":"8e55ee84-d422-4423-8614-3ebed0c0941d","Type":"ContainerStarted","Data":"4b43a623d28d8975409c85b5883b5a5c4b8b1e32e01e4bf26e400110cd7fac56"}
Jul 13 21:59:32.846825 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:32.846779    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4" podStartSLOduration=15.846766406 podStartE2EDuration="15.846766406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:32.846252181 +0000 UTC m=+1814.002646022" watchObservedRunningTime="2026-07-13 21:59:32.846766406 +0000 UTC m=+1814.003160247"
Jul 13 21:59:33.829107 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:33.829070    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerID="4b43a623d28d8975409c85b5883b5a5c4b8b1e32e01e4bf26e400110cd7fac56" exitCode=0
Jul 13 21:59:33.829564 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:33.829144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4" event={"ID":"8e55ee84-d422-4423-8614-3ebed0c0941d","Type":"ContainerDied","Data":"4b43a623d28d8975409c85b5883b5a5c4b8b1e32e01e4bf26e400110cd7fac56"}
Jul 13 21:59:33.829564 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:33.829237    2582 scope.go:117] "RemoveContainer" containerID="041f90eccd1aadfc31c9e9a19a4e176ba471c1b102e3f4605fb4f000f40efe73"
Jul 13 21:59:33.873942 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:33.873911    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"]
Jul 13 21:59:34.966796 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:34.966767    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"
Jul 13 21:59:35.063367 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:35.063332    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e55ee84-d422-4423-8614-3ebed0c0941d-maas-api-service-ca\") pod \"8e55ee84-d422-4423-8614-3ebed0c0941d\" (UID: \"8e55ee84-d422-4423-8614-3ebed0c0941d\") "
Jul 13 21:59:35.063669 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:35.063647    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e55ee84-d422-4423-8614-3ebed0c0941d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e55ee84-d422-4423-8614-3ebed0c0941d" (UID: "8e55ee84-d422-4423-8614-3ebed0c0941d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:59:35.164929 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:35.164863    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e55ee84-d422-4423-8614-3ebed0c0941d-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 21:59:35.842025 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:35.841992    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"
Jul 13 21:59:35.842025 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:35.842017    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4" event={"ID":"8e55ee84-d422-4423-8614-3ebed0c0941d","Type":"ContainerDied","Data":"f29b9695e92b16353b05f8ce59c7d3d02b08a2ddde9de876f584f99a65b1a73a"}
Jul 13 21:59:35.842321 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:35.842066    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f29b9695e92b16353b05f8ce59c7d3d02b08a2ddde9de876f584f99a65b1a73a"
Jul 13 21:59:35.877471 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:35.877444    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"]
Jul 13 21:59:35.881947 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:35.881924    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-bwjj4"]
Jul 13 21:59:37.362902 ip-10-0-140-193 kubenswrapper[2582]: I0713 21:59:37.362871    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d" path="/var/lib/kubelet/pods/8e55ee84-d422-4423-8614-3ebed0c0941d/volumes"
Jul 13 22:00:00.149636 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.149595    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj"]
Jul 13 22:00:00.150770 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.150744    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerName="revoke-keys"
Jul 13 22:00:00.150770 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.150768    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerName="revoke-keys"
Jul 13 22:00:00.150920 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.150777    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerName="revoke-keys"
Jul 13 22:00:00.150920 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.150783    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerName="revoke-keys"
Jul 13 22:00:00.150920 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.150791    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerName="revoke-keys"
Jul 13 22:00:00.150920 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.150796    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerName="revoke-keys"
Jul 13 22:00:00.150920 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.150858    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerName="revoke-keys"
Jul 13 22:00:00.150920 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.150866    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerName="revoke-keys"
Jul 13 22:00:00.153766 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.153739    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn"]
Jul 13 22:00:00.153897 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.153839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj"
Jul 13 22:00:00.154633 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.154604    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e55ee84-d422-4423-8614-3ebed0c0941d" containerName="revoke-keys"
Jul 13 22:00:00.156715 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.156690    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:00:00.157619 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.157597    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj"]
Jul 13 22:00:00.157736 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.157638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn"]
Jul 13 22:00:00.157736 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.157716    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn"
Jul 13 22:00:00.166690 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.166673    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj"
Jul 13 22:00:00.173330 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.173310    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn"
Jul 13 22:00:00.326767 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.326742    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn"]
Jul 13 22:00:00.329131 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:00:00.329106    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33f5af5_6a1f_4cd0_8d24_ca295a4b0f6c.slice/crio-666b803916f86e22b686e6cd90d2f31fcb914e09c5a58e2dce870ba93934fbd7 WatchSource:0}: Error finding container 666b803916f86e22b686e6cd90d2f31fcb914e09c5a58e2dce870ba93934fbd7: Status 404 returned error can't find the container with id 666b803916f86e22b686e6cd90d2f31fcb914e09c5a58e2dce870ba93934fbd7
Jul 13 22:00:00.514760 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.514736    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj"]
Jul 13 22:00:00.517828 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:00:00.517800    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db382e4_e8a8_47d7_81d8_a0fb15ed3044.slice/crio-62d58b6d31145968fbb64e91e30d7d90883d55d9936a16c1c82f58323dba45c2 WatchSource:0}: Error finding container 62d58b6d31145968fbb64e91e30d7d90883d55d9936a16c1c82f58323dba45c2: Status 404 returned error can't find the container with id 62d58b6d31145968fbb64e91e30d7d90883d55d9936a16c1c82f58323dba45c2
Jul 13 22:00:00.959732 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.959697    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn" event={"ID":"b33f5af5-6a1f-4cd0-8d24-ca295a4b0f6c","Type":"ContainerStarted","Data":"0badaa679ae81a9e855fd95be97b4816db6b06d2c0c3777138adf41392b0c88a"}
Jul 13 22:00:00.959881 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.959760    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn" event={"ID":"b33f5af5-6a1f-4cd0-8d24-ca295a4b0f6c","Type":"ContainerStarted","Data":"666b803916f86e22b686e6cd90d2f31fcb914e09c5a58e2dce870ba93934fbd7"}
Jul 13 22:00:00.961368 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.961343    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj" event={"ID":"4db382e4-e8a8-47d7-81d8-a0fb15ed3044","Type":"ContainerStarted","Data":"2940fc7aa5f8be7f25e09c9d33277c8823f97f557dcdbe68a946025b02083d19"}
Jul 13 22:00:00.961476 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.961386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj" event={"ID":"4db382e4-e8a8-47d7-81d8-a0fb15ed3044","Type":"ContainerStarted","Data":"62d58b6d31145968fbb64e91e30d7d90883d55d9936a16c1c82f58323dba45c2"}
Jul 13 22:00:00.980359 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.980317    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn" podStartSLOduration=0.98029956 podStartE2EDuration="980.29956ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:00.978293518 +0000 UTC m=+1842.134687352" watchObservedRunningTime="2026-07-13 22:00:00.98029956 +0000 UTC m=+1842.136693401"
Jul 13 22:00:00.998851 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:00.998810    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj" podStartSLOduration=0.998798852 podStartE2EDuration="998.798852ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:00.997823666 +0000 UTC m=+1842.154217509" watchObservedRunningTime="2026-07-13 22:00:00.998798852 +0000 UTC m=+1842.155192693"
Jul 13 22:00:01.967842 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:01.967750    2582 generic.go:356] "Generic (PLEG): container finished" podID="b33f5af5-6a1f-4cd0-8d24-ca295a4b0f6c" containerID="0badaa679ae81a9e855fd95be97b4816db6b06d2c0c3777138adf41392b0c88a" exitCode=0
Jul 13 22:00:01.968275 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:01.967841    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn" event={"ID":"b33f5af5-6a1f-4cd0-8d24-ca295a4b0f6c","Type":"ContainerDied","Data":"0badaa679ae81a9e855fd95be97b4816db6b06d2c0c3777138adf41392b0c88a"}
Jul 13 22:00:01.969305 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:01.969284    2582 generic.go:356] "Generic (PLEG): container finished" podID="4db382e4-e8a8-47d7-81d8-a0fb15ed3044" containerID="2940fc7aa5f8be7f25e09c9d33277c8823f97f557dcdbe68a946025b02083d19" exitCode=0
Jul 13 22:00:01.969399 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:01.969375    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj" event={"ID":"4db382e4-e8a8-47d7-81d8-a0fb15ed3044","Type":"ContainerDied","Data":"2940fc7aa5f8be7f25e09c9d33277c8823f97f557dcdbe68a946025b02083d19"}
Jul 13 22:00:03.109722 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:03.109693    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn"
Jul 13 22:00:03.142714 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:03.142687    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj"
Jul 13 22:00:03.980369 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:03.980339    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn"
Jul 13 22:00:03.980578 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:03.980341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733000-kxznn" event={"ID":"b33f5af5-6a1f-4cd0-8d24-ca295a4b0f6c","Type":"ContainerDied","Data":"666b803916f86e22b686e6cd90d2f31fcb914e09c5a58e2dce870ba93934fbd7"}
Jul 13 22:00:03.980578 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:03.980517    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="666b803916f86e22b686e6cd90d2f31fcb914e09c5a58e2dce870ba93934fbd7"
Jul 13 22:00:03.981801 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:03.981779    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj"
Jul 13 22:00:03.981940 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:03.981784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj" event={"ID":"4db382e4-e8a8-47d7-81d8-a0fb15ed3044","Type":"ContainerDied","Data":"62d58b6d31145968fbb64e91e30d7d90883d55d9936a16c1c82f58323dba45c2"}
Jul 13 22:00:03.981940 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:03.981881    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62d58b6d31145968fbb64e91e30d7d90883d55d9936a16c1c82f58323dba45c2"
Jul 13 22:00:07.194858 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.194816    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh"]
Jul 13 22:00:07.195346 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.195327    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33f5af5-6a1f-4cd0-8d24-ca295a4b0f6c" containerName="cleanup"
Jul 13 22:00:07.195438 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.195349    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33f5af5-6a1f-4cd0-8d24-ca295a4b0f6c" containerName="cleanup"
Jul 13 22:00:07.195438 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.195364    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db382e4-e8a8-47d7-81d8-a0fb15ed3044" containerName="cleanup"
Jul 13 22:00:07.195438 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.195373    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db382e4-e8a8-47d7-81d8-a0fb15ed3044" containerName="cleanup"
Jul 13 22:00:07.195582 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.195488    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33f5af5-6a1f-4cd0-8d24-ca295a4b0f6c" containerName="cleanup"
Jul 13 22:00:07.195582 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.195504    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db382e4-e8a8-47d7-81d8-a0fb15ed3044" containerName="cleanup"
Jul 13 22:00:07.198564 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.198540    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh"
Jul 13 22:00:07.201431 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.201403    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:00:07.207265 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.207235    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh"]
Jul 13 22:00:07.343265 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.343223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a58d-9860-474e-9b74-25f77113d5fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e874d59-569xh\" (UID: \"71c8a58d-9860-474e-9b74-25f77113d5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh"
Jul 13 22:00:07.444080 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.444049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a58d-9860-474e-9b74-25f77113d5fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e874d59-569xh\" (UID: \"71c8a58d-9860-474e-9b74-25f77113d5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh"
Jul 13 22:00:07.444714 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.444695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a58d-9860-474e-9b74-25f77113d5fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7e874d59-569xh\" (UID: \"71c8a58d-9860-474e-9b74-25f77113d5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh"
Jul 13 22:00:07.511525 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.511497    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh"
Jul 13 22:00:07.638902 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:07.638878    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh"]
Jul 13 22:00:07.641177 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:00:07.641128    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c8a58d_9860_474e_9b74_25f77113d5fd.slice/crio-7b96c612810a85d3a5b44807d5b236d897b1b2bc795620f323ad84f6b07d16de WatchSource:0}: Error finding container 7b96c612810a85d3a5b44807d5b236d897b1b2bc795620f323ad84f6b07d16de: Status 404 returned error can't find the container with id 7b96c612810a85d3a5b44807d5b236d897b1b2bc795620f323ad84f6b07d16de
Jul 13 22:00:08.001911 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:08.001872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh" event={"ID":"71c8a58d-9860-474e-9b74-25f77113d5fd","Type":"ContainerStarted","Data":"0e7619ab0c4aa96c9fc37ad07ed3e74a6baf44a8b10da6fe8540d0370da64b88"}
Jul 13 22:00:08.002115 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:08.001933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh" event={"ID":"71c8a58d-9860-474e-9b74-25f77113d5fd","Type":"ContainerStarted","Data":"7b96c612810a85d3a5b44807d5b236d897b1b2bc795620f323ad84f6b07d16de"}
Jul 13 22:00:08.018258 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:08.018202    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh" podStartSLOduration=1.01818288 podStartE2EDuration="1.01818288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:08.017518303 +0000 UTC m=+1849.173912149" watchObservedRunningTime="2026-07-13 22:00:08.01818288 +0000 UTC m=+1849.174576729"
Jul 13 22:00:09.007799 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:09.007768    2582 generic.go:356] "Generic (PLEG): container finished" podID="71c8a58d-9860-474e-9b74-25f77113d5fd" containerID="0e7619ab0c4aa96c9fc37ad07ed3e74a6baf44a8b10da6fe8540d0370da64b88" exitCode=0
Jul 13 22:00:09.008219 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:09.007846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh" event={"ID":"71c8a58d-9860-474e-9b74-25f77113d5fd","Type":"ContainerDied","Data":"0e7619ab0c4aa96c9fc37ad07ed3e74a6baf44a8b10da6fe8540d0370da64b88"}
Jul 13 22:00:10.136530 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:10.136506    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh"
Jul 13 22:00:10.170824 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:10.170799    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a58d-9860-474e-9b74-25f77113d5fd-maas-api-service-ca\") pod \"71c8a58d-9860-474e-9b74-25f77113d5fd\" (UID: \"71c8a58d-9860-474e-9b74-25f77113d5fd\") "
Jul 13 22:00:10.171106 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:10.171084    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c8a58d-9860-474e-9b74-25f77113d5fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71c8a58d-9860-474e-9b74-25f77113d5fd" (UID: "71c8a58d-9860-474e-9b74-25f77113d5fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:00:10.271828 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:10.271763    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a58d-9860-474e-9b74-25f77113d5fd-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:00:11.018185 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:11.018137    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh"
Jul 13 22:00:11.018185 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:11.018174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7e874d59-569xh" event={"ID":"71c8a58d-9860-474e-9b74-25f77113d5fd","Type":"ContainerDied","Data":"7b96c612810a85d3a5b44807d5b236d897b1b2bc795620f323ad84f6b07d16de"}
Jul 13 22:00:11.018399 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:11.018227    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b96c612810a85d3a5b44807d5b236d897b1b2bc795620f323ad84f6b07d16de"
Jul 13 22:00:17.313529 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:17.313485    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:17.428872 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:17.428831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:17.798720 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:17.798685    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"]
Jul 13 22:00:17.799049 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:17.799019    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh" podUID="79f9b815-69aa-4a4c-9d59-5affc5588fda" containerName="maas-api" containerID="cri-o://3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51" gracePeriod=30
Jul 13 22:00:17.832291 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:17.832261    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj"]
Jul 13 22:00:17.836293 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:17.836271    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-7e874d59-29733000-8wtcj"]
Jul 13 22:00:17.844669 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:17.844650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:18.040918 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.040891    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 22:00:18.055418 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.055357    2582 generic.go:356] "Generic (PLEG): container finished" podID="79f9b815-69aa-4a4c-9d59-5affc5588fda" containerID="3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51" exitCode=0
Jul 13 22:00:18.055418 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.055397    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"
Jul 13 22:00:18.055587 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.055447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh" event={"ID":"79f9b815-69aa-4a4c-9d59-5affc5588fda","Type":"ContainerDied","Data":"3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51"}
Jul 13 22:00:18.055587 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.055507    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh" event={"ID":"79f9b815-69aa-4a4c-9d59-5affc5588fda","Type":"ContainerDied","Data":"d51f7468360d5cb78a8f86d3e180b5719c28df394593742249be83c0e815511d"}
Jul 13 22:00:18.055587 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.055536    2582 scope.go:117] "RemoveContainer" containerID="3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51"
Jul 13 22:00:18.064829 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.064808    2582 scope.go:117] "RemoveContainer" containerID="3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51"
Jul 13 22:00:18.065185 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:00:18.065126    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51\": container with ID starting with 3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51 not found: ID does not exist" containerID="3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51"
Jul 13 22:00:18.065303 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.065230    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51"} err="failed to get container status \"3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51\": rpc error: code = NotFound desc = could not find container \"3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51\": container with ID starting with 3b985e2e8f6aedceb641ee3451fcc464d4cf4beea465b9ae582720387f1e6d51 not found: ID does not exist"
Jul 13 22:00:18.134694 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.134675    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f9b815-69aa-4a4c-9d59-5affc5588fda-openshift-service-ca\") pod \"79f9b815-69aa-4a4c-9d59-5affc5588fda\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") "
Jul 13 22:00:18.134793 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.134749    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f9b815-69aa-4a4c-9d59-5affc5588fda-maas-api-tls\") pod \"79f9b815-69aa-4a4c-9d59-5affc5588fda\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") "
Jul 13 22:00:18.134793 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.134774    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzfqh\" (UniqueName: \"kubernetes.io/projected/79f9b815-69aa-4a4c-9d59-5affc5588fda-kube-api-access-fzfqh\") pod \"79f9b815-69aa-4a4c-9d59-5affc5588fda\" (UID: \"79f9b815-69aa-4a4c-9d59-5affc5588fda\") "
Jul 13 22:00:18.135026 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.135005    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f9b815-69aa-4a4c-9d59-5affc5588fda-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79f9b815-69aa-4a4c-9d59-5affc5588fda" (UID: "79f9b815-69aa-4a4c-9d59-5affc5588fda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:00:18.136767 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.136740    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f9b815-69aa-4a4c-9d59-5affc5588fda-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79f9b815-69aa-4a4c-9d59-5affc5588fda" (UID: "79f9b815-69aa-4a4c-9d59-5affc5588fda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:00:18.136855 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.136771    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f9b815-69aa-4a4c-9d59-5affc5588fda-kube-api-access-fzfqh" (OuterVolumeSpecName: "kube-api-access-fzfqh") pod "79f9b815-69aa-4a4c-9d59-5affc5588fda" (UID: "79f9b815-69aa-4a4c-9d59-5affc5588fda"). InnerVolumeSpecName "kube-api-access-fzfqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:00:18.236035 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.236010    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f9b815-69aa-4a4c-9d59-5affc5588fda-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:00:18.236035 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.236034    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f9b815-69aa-4a4c-9d59-5affc5588fda-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:00:18.236224 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.236045    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzfqh\" (UniqueName: \"kubernetes.io/projected/79f9b815-69aa-4a4c-9d59-5affc5588fda-kube-api-access-fzfqh\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:00:18.380133 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.380106    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"]
Jul 13 22:00:18.387846 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:18.387825    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e874d59-75b5f6c588-hstzh"]
Jul 13 22:00:19.363105 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:19.363075    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db382e4-e8a8-47d7-81d8-a0fb15ed3044" path="/var/lib/kubelet/pods/4db382e4-e8a8-47d7-81d8-a0fb15ed3044/volumes"
Jul 13 22:00:19.363445 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:19.363430    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79f9b815-69aa-4a4c-9d59-5affc5588fda" path="/var/lib/kubelet/pods/79f9b815-69aa-4a4c-9d59-5affc5588fda/volumes"
Jul 13 22:00:20.054558 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:20.054523    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:00:23.452027 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:23.451984    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:23.554291 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:23.554255    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:23.703953 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:23.703866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:23.805111 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:23.805076    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:23.899954 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:23.899912    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:24.936739 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:24.936682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:25.075715 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.075687    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"]
Jul 13 22:00:25.076036 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.076023    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c8a58d-9860-474e-9b74-25f77113d5fd" containerName="revoke-keys"
Jul 13 22:00:25.076089 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.076038    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c8a58d-9860-474e-9b74-25f77113d5fd" containerName="revoke-keys"
Jul 13 22:00:25.076089 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.076059    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f9b815-69aa-4a4c-9d59-5affc5588fda" containerName="maas-api"
Jul 13 22:00:25.076089 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.076065    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f9b815-69aa-4a4c-9d59-5affc5588fda" containerName="maas-api"
Jul 13 22:00:25.076207 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.076114    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f9b815-69aa-4a4c-9d59-5affc5588fda" containerName="maas-api"
Jul 13 22:00:25.076207 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.076122    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c8a58d-9860-474e-9b74-25f77113d5fd" containerName="revoke-keys"
Jul 13 22:00:25.078985 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.078964    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.081934 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.081907    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 22:00:25.082046 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.081958    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kfxsg\""
Jul 13 22:00:25.082680 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.082664    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-30bfd308-serving-cert\""
Jul 13 22:00:25.083280 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.083265    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 22:00:25.105621 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.105601    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"]
Jul 13 22:00:25.151083 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.151050    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:25.198518 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.198456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71ba89e5-20f0-4f05-840c-88a8933f3616-maas-api-tls\") pod \"maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.198518 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.198484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ba89e5-20f0-4f05-840c-88a8933f3616-openshift-service-ca\") pod \"maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.198639 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.198562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9lk\" (UniqueName: \"kubernetes.io/projected/71ba89e5-20f0-4f05-840c-88a8933f3616-kube-api-access-ph9lk\") pod \"maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.289418 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.289395    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:25.299410 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.299388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9lk\" (UniqueName: \"kubernetes.io/projected/71ba89e5-20f0-4f05-840c-88a8933f3616-kube-api-access-ph9lk\") pod \"maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.299488 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.299453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71ba89e5-20f0-4f05-840c-88a8933f3616-maas-api-tls\") pod \"maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.299488 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.299474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ba89e5-20f0-4f05-840c-88a8933f3616-openshift-service-ca\") pod \"maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.300082 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.300063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ba89e5-20f0-4f05-840c-88a8933f3616-openshift-service-ca\") pod \"maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.301825 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.301808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71ba89e5-20f0-4f05-840c-88a8933f3616-maas-api-tls\") pod \"maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.311159 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.311115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9lk\" (UniqueName: \"kubernetes.io/projected/71ba89e5-20f0-4f05-840c-88a8933f3616-kube-api-access-ph9lk\") pod \"maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.389253 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.389234    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:25.524729 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.524704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"]
Jul 13 22:00:25.526725 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:00:25.526699    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ba89e5_20f0_4f05_840c_88a8933f3616.slice/crio-e5c96ef14d272947704952a150c558f0c5ff4b53904c0baac5c67ed78af5ef19 WatchSource:0}: Error finding container e5c96ef14d272947704952a150c558f0c5ff4b53904c0baac5c67ed78af5ef19: Status 404 returned error can't find the container with id e5c96ef14d272947704952a150c558f0c5ff4b53904c0baac5c67ed78af5ef19
Jul 13 22:00:25.572470 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.572440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:25.797871 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.797819    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:25.892370 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:25.892339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:26.093962 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:26.093882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j" event={"ID":"71ba89e5-20f0-4f05-840c-88a8933f3616","Type":"ContainerStarted","Data":"e5c96ef14d272947704952a150c558f0c5ff4b53904c0baac5c67ed78af5ef19"}
Jul 13 22:00:27.100301 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:27.100269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j" event={"ID":"71ba89e5-20f0-4f05-840c-88a8933f3616","Type":"ContainerStarted","Data":"51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b"}
Jul 13 22:00:27.100685 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:27.100401    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:27.119321 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:27.119270    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j" podStartSLOduration=1.342537689 podStartE2EDuration="2.119250767s" totalImagesPullingTime="776.713078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:27.118795254 +0000 UTC m=+1868.275189095" watchObservedRunningTime="2026-07-13 22:00:27.119250767 +0000 UTC m=+1868.275644608"
Jul 13 22:00:27.485058 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:27.485005    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:00:29.929190 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:29.929146    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"]
Jul 13 22:00:29.932604 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:29.932588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"
Jul 13 22:00:29.935315 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:29.935295    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:00:29.940664 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:29.940639    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"]
Jul 13 22:00:30.038992 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:30.038964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bf3071-ba3f-48cc-8788-ff466ccd6383-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n\" (UID: \"e0bf3071-ba3f-48cc-8788-ff466ccd6383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"
Jul 13 22:00:30.139535 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:30.139506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bf3071-ba3f-48cc-8788-ff466ccd6383-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n\" (UID: \"e0bf3071-ba3f-48cc-8788-ff466ccd6383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"
Jul 13 22:00:30.140137 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:30.140108    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bf3071-ba3f-48cc-8788-ff466ccd6383-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n\" (UID: \"e0bf3071-ba3f-48cc-8788-ff466ccd6383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"
Jul 13 22:00:30.243308 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:30.243286    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"
Jul 13 22:00:30.379784 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:30.379760    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"]
Jul 13 22:00:30.382020 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:00:30.381993    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bf3071_ba3f_48cc_8788_ff466ccd6383.slice/crio-1b92156dbd095b7ceb2edd5bce982f9c0fbe475456ea63c1550c6f138e528d3e WatchSource:0}: Error finding container 1b92156dbd095b7ceb2edd5bce982f9c0fbe475456ea63c1550c6f138e528d3e: Status 404 returned error can't find the container with id 1b92156dbd095b7ceb2edd5bce982f9c0fbe475456ea63c1550c6f138e528d3e
Jul 13 22:00:31.119803 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:31.119772    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerID="516fb48b4ecd51453521eeeb9f2b9728937e25ee14ea3b29b313b32764528959" exitCode=7
Jul 13 22:00:31.120105 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:31.119834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n" event={"ID":"e0bf3071-ba3f-48cc-8788-ff466ccd6383","Type":"ContainerDied","Data":"516fb48b4ecd51453521eeeb9f2b9728937e25ee14ea3b29b313b32764528959"}
Jul 13 22:00:31.120105 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:31.119886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n" event={"ID":"e0bf3071-ba3f-48cc-8788-ff466ccd6383","Type":"ContainerStarted","Data":"1b92156dbd095b7ceb2edd5bce982f9c0fbe475456ea63c1550c6f138e528d3e"}
Jul 13 22:00:31.120105 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:31.120034    2582 scope.go:117] "RemoveContainer" containerID="516fb48b4ecd51453521eeeb9f2b9728937e25ee14ea3b29b313b32764528959"
Jul 13 22:00:32.126171 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:32.126072    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerID="401f43d47fd1a40f89178904160dba6126cd2a3adf6a86d8539077110d4f318d" exitCode=7
Jul 13 22:00:32.126611 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:32.126176    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n" event={"ID":"e0bf3071-ba3f-48cc-8788-ff466ccd6383","Type":"ContainerDied","Data":"401f43d47fd1a40f89178904160dba6126cd2a3adf6a86d8539077110d4f318d"}
Jul 13 22:00:32.126611 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:32.126233    2582 scope.go:117] "RemoveContainer" containerID="516fb48b4ecd51453521eeeb9f2b9728937e25ee14ea3b29b313b32764528959"
Jul 13 22:00:32.126611 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:32.126351    2582 scope.go:117] "RemoveContainer" containerID="401f43d47fd1a40f89178904160dba6126cd2a3adf6a86d8539077110d4f318d"
Jul 13 22:00:32.126611 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:00:32.126571    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n_odh-ai-gateway-infra(e0bf3071-ba3f-48cc-8788-ff466ccd6383)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383"
Jul 13 22:00:33.059300 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:33.055412    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:00:33.109861 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:33.109830    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:00:33.132973 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:33.132951    2582 scope.go:117] "RemoveContainer" containerID="401f43d47fd1a40f89178904160dba6126cd2a3adf6a86d8539077110d4f318d"
Jul 13 22:00:33.133326 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:00:33.133105    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n_odh-ai-gateway-infra(e0bf3071-ba3f-48cc-8788-ff466ccd6383)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383"
Jul 13 22:00:43.358786 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:43.358749    2582 scope.go:117] "RemoveContainer" containerID="401f43d47fd1a40f89178904160dba6126cd2a3adf6a86d8539077110d4f318d"
Jul 13 22:00:44.185537 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:44.185508    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerID="30b954d08781012567f180a91068bae7289c16acf649c9584e9608906a6d2c3a" exitCode=0
Jul 13 22:00:44.185679 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:44.185579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n" event={"ID":"e0bf3071-ba3f-48cc-8788-ff466ccd6383","Type":"ContainerDied","Data":"30b954d08781012567f180a91068bae7289c16acf649c9584e9608906a6d2c3a"}
Jul 13 22:00:44.185679 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:44.185643    2582 scope.go:117] "RemoveContainer" containerID="401f43d47fd1a40f89178904160dba6126cd2a3adf6a86d8539077110d4f318d"
Jul 13 22:00:45.217923 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:45.217880    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"]
Jul 13 22:00:45.326416 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:45.326390    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"
Jul 13 22:00:45.464529 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:45.464456    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bf3071-ba3f-48cc-8788-ff466ccd6383-maas-api-service-ca\") pod \"e0bf3071-ba3f-48cc-8788-ff466ccd6383\" (UID: \"e0bf3071-ba3f-48cc-8788-ff466ccd6383\") "
Jul 13 22:00:45.464793 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:45.464770    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0bf3071-ba3f-48cc-8788-ff466ccd6383-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0bf3071-ba3f-48cc-8788-ff466ccd6383" (UID: "e0bf3071-ba3f-48cc-8788-ff466ccd6383"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:00:45.565530 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:45.565504    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bf3071-ba3f-48cc-8788-ff466ccd6383-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:00:46.197617 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:46.197578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n" event={"ID":"e0bf3071-ba3f-48cc-8788-ff466ccd6383","Type":"ContainerDied","Data":"1b92156dbd095b7ceb2edd5bce982f9c0fbe475456ea63c1550c6f138e528d3e"}
Jul 13 22:00:46.197617 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:46.197618    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"
Jul 13 22:00:46.197817 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:46.197640    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b92156dbd095b7ceb2edd5bce982f9c0fbe475456ea63c1550c6f138e528d3e"
Jul 13 22:00:46.228881 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:46.228855    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"]
Jul 13 22:00:46.232669 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:46.232645    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-q9f8n"]
Jul 13 22:00:47.363158 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:00:47.363115    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" path="/var/lib/kubelet/pods/e0bf3071-ba3f-48cc-8788-ff466ccd6383/volumes"
Jul 13 22:01:03.319889 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:03.319856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:03.420371 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:03.420329    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:19.974096 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.974060    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv"]
Jul 13 22:01:19.974490 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.974449    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerName="revoke-keys"
Jul 13 22:01:19.974490 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.974466    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerName="revoke-keys"
Jul 13 22:01:19.974490 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.974489    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerName="revoke-keys"
Jul 13 22:01:19.974601 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.974495    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerName="revoke-keys"
Jul 13 22:01:19.974601 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.974554    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerName="revoke-keys"
Jul 13 22:01:19.974601 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.974563    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerName="revoke-keys"
Jul 13 22:01:19.974601 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.974569    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerName="revoke-keys"
Jul 13 22:01:19.977797 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.977781    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv"
Jul 13 22:01:19.980561 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.980532    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:01:19.986432 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:19.986412    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv"]
Jul 13 22:01:20.043442 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:20.043412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a337783-6294-4eb9-a439-886c7f7cd349-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30bfd308-d66pv\" (UID: \"6a337783-6294-4eb9-a439-886c7f7cd349\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv"
Jul 13 22:01:20.144542 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:20.144509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a337783-6294-4eb9-a439-886c7f7cd349-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30bfd308-d66pv\" (UID: \"6a337783-6294-4eb9-a439-886c7f7cd349\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv"
Jul 13 22:01:20.145124 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:20.145103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a337783-6294-4eb9-a439-886c7f7cd349-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30bfd308-d66pv\" (UID: \"6a337783-6294-4eb9-a439-886c7f7cd349\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv"
Jul 13 22:01:20.289573 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:20.289542    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv"
Jul 13 22:01:20.417930 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:20.417903    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv"]
Jul 13 22:01:20.420108 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:01:20.420077    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a337783_6294_4eb9_a439_886c7f7cd349.slice/crio-ad443c1790fb3d44e045453c9a438e1740e08febaad9ac5edd19108f0bf11d44 WatchSource:0}: Error finding container ad443c1790fb3d44e045453c9a438e1740e08febaad9ac5edd19108f0bf11d44: Status 404 returned error can't find the container with id ad443c1790fb3d44e045453c9a438e1740e08febaad9ac5edd19108f0bf11d44
Jul 13 22:01:20.515472 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:20.515446    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:20.613394 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:20.613328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:21.367740 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:21.367666    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a337783-6294-4eb9-a439-886c7f7cd349" containerID="7a8b40443ed635e14641e8870965984aff1641abdb2d920c838d646966130a41" exitCode=0
Jul 13 22:01:21.368244 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:21.367749    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv" event={"ID":"6a337783-6294-4eb9-a439-886c7f7cd349","Type":"ContainerDied","Data":"7a8b40443ed635e14641e8870965984aff1641abdb2d920c838d646966130a41"}
Jul 13 22:01:21.368244 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:21.367804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv" event={"ID":"6a337783-6294-4eb9-a439-886c7f7cd349","Type":"ContainerStarted","Data":"ad443c1790fb3d44e045453c9a438e1740e08febaad9ac5edd19108f0bf11d44"}
Jul 13 22:01:22.496480 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:22.496458    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv"
Jul 13 22:01:22.568550 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:22.568517    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a337783-6294-4eb9-a439-886c7f7cd349-maas-api-service-ca\") pod \"6a337783-6294-4eb9-a439-886c7f7cd349\" (UID: \"6a337783-6294-4eb9-a439-886c7f7cd349\") "
Jul 13 22:01:22.568862 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:22.568839    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a337783-6294-4eb9-a439-886c7f7cd349-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a337783-6294-4eb9-a439-886c7f7cd349" (UID: "6a337783-6294-4eb9-a439-886c7f7cd349"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:01:22.669165 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:22.669081    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a337783-6294-4eb9-a439-886c7f7cd349-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:01:23.377134 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:23.377098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv" event={"ID":"6a337783-6294-4eb9-a439-886c7f7cd349","Type":"ContainerDied","Data":"ad443c1790fb3d44e045453c9a438e1740e08febaad9ac5edd19108f0bf11d44"}
Jul 13 22:01:23.377312 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:23.377110    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30bfd308-d66pv"
Jul 13 22:01:23.377312 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:23.377176    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad443c1790fb3d44e045453c9a438e1740e08febaad9ac5edd19108f0bf11d44"
Jul 13 22:01:30.488501 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.486283    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"]
Jul 13 22:01:30.488501 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.486510    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j" podUID="71ba89e5-20f0-4f05-840c-88a8933f3616" containerName="maas-api" containerID="cri-o://51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b" gracePeriod=30
Jul 13 22:01:30.524546 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.524508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:30.526111 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:01:30.526081    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ba89e5_20f0_4f05_840c_88a8933f3616.slice/crio-51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b.scope\": RecentStats: unable to find data in memory cache]"
Jul 13 22:01:30.735003 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.734977    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:01:30.844609 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.844533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71ba89e5-20f0-4f05-840c-88a8933f3616-maas-api-tls\") pod \"71ba89e5-20f0-4f05-840c-88a8933f3616\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") "
Jul 13 22:01:30.844609 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.844568    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph9lk\" (UniqueName: \"kubernetes.io/projected/71ba89e5-20f0-4f05-840c-88a8933f3616-kube-api-access-ph9lk\") pod \"71ba89e5-20f0-4f05-840c-88a8933f3616\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") "
Jul 13 22:01:30.844780 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.844612    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ba89e5-20f0-4f05-840c-88a8933f3616-openshift-service-ca\") pod \"71ba89e5-20f0-4f05-840c-88a8933f3616\" (UID: \"71ba89e5-20f0-4f05-840c-88a8933f3616\") "
Jul 13 22:01:30.845038 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.845014    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ba89e5-20f0-4f05-840c-88a8933f3616-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71ba89e5-20f0-4f05-840c-88a8933f3616" (UID: "71ba89e5-20f0-4f05-840c-88a8933f3616"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:01:30.846672 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.846642    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ba89e5-20f0-4f05-840c-88a8933f3616-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71ba89e5-20f0-4f05-840c-88a8933f3616" (UID: "71ba89e5-20f0-4f05-840c-88a8933f3616"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:01:30.846672 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.846646    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ba89e5-20f0-4f05-840c-88a8933f3616-kube-api-access-ph9lk" (OuterVolumeSpecName: "kube-api-access-ph9lk") pod "71ba89e5-20f0-4f05-840c-88a8933f3616" (UID: "71ba89e5-20f0-4f05-840c-88a8933f3616"). InnerVolumeSpecName "kube-api-access-ph9lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:01:30.945383 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.945345    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71ba89e5-20f0-4f05-840c-88a8933f3616-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:01:30.945383 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.945380    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph9lk\" (UniqueName: \"kubernetes.io/projected/71ba89e5-20f0-4f05-840c-88a8933f3616-kube-api-access-ph9lk\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:01:30.945383 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:30.945391    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ba89e5-20f0-4f05-840c-88a8933f3616-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:01:31.412617 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:31.412585    2582 generic.go:356] "Generic (PLEG): container finished" podID="71ba89e5-20f0-4f05-840c-88a8933f3616" containerID="51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b" exitCode=0
Jul 13 22:01:31.412848 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:31.412627    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"
Jul 13 22:01:31.412848 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:31.412629    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j" event={"ID":"71ba89e5-20f0-4f05-840c-88a8933f3616","Type":"ContainerDied","Data":"51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b"}
Jul 13 22:01:31.412848 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:31.412779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j" event={"ID":"71ba89e5-20f0-4f05-840c-88a8933f3616","Type":"ContainerDied","Data":"e5c96ef14d272947704952a150c558f0c5ff4b53904c0baac5c67ed78af5ef19"}
Jul 13 22:01:31.412848 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:31.412817    2582 scope.go:117] "RemoveContainer" containerID="51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b"
Jul 13 22:01:31.421719 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:31.421689    2582 scope.go:117] "RemoveContainer" containerID="51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b"
Jul 13 22:01:31.421971 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:01:31.421955    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b\": container with ID starting with 51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b not found: ID does not exist" containerID="51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b"
Jul 13 22:01:31.422018 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:31.421983    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b"} err="failed to get container status \"51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b\": rpc error: code = NotFound desc = could not find container \"51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b\": container with ID starting with 51aea3a0dafb2e46aef02de65178964afc2cccd7c633a762d50560b1774ba10b not found: ID does not exist"
Jul 13 22:01:31.429887 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:31.429864    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"]
Jul 13 22:01:31.436132 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:31.436109    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30bfd308-7d6b9c9d6-gb27j"]
Jul 13 22:01:33.362702 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:33.362667    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ba89e5-20f0-4f05-840c-88a8933f3616" path="/var/lib/kubelet/pods/71ba89e5-20f0-4f05-840c-88a8933f3616/volumes"
Jul 13 22:01:33.554029 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:33.553994    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:01:35.064415 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:35.064366    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:41.229283 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:41.229241    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:41.330549 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:41.330517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:46.123191 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:46.123134    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:46.450723 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:46.450688    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:47.640439 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.640401    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:47.703884 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.703847    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"]
Jul 13 22:01:47.704445 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.704418    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ba89e5-20f0-4f05-840c-88a8933f3616" containerName="maas-api"
Jul 13 22:01:47.704445 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.704444    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ba89e5-20f0-4f05-840c-88a8933f3616" containerName="maas-api"
Jul 13 22:01:47.704644 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.704473    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a337783-6294-4eb9-a439-886c7f7cd349" containerName="revoke-keys"
Jul 13 22:01:47.704644 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.704482    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a337783-6294-4eb9-a439-886c7f7cd349" containerName="revoke-keys"
Jul 13 22:01:47.704644 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.704505    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerName="revoke-keys"
Jul 13 22:01:47.704644 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.704542    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bf3071-ba3f-48cc-8788-ff466ccd6383" containerName="revoke-keys"
Jul 13 22:01:47.704644 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.704644    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a337783-6294-4eb9-a439-886c7f7cd349" containerName="revoke-keys"
Jul 13 22:01:47.704900 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.704660    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ba89e5-20f0-4f05-840c-88a8933f3616" containerName="maas-api"
Jul 13 22:01:47.709111 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.709090    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:47.711694 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.711672    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-628bb6aa-serving-cert\""
Jul 13 22:01:47.711996 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.711974    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 22:01:47.712085 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.712013    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kfxsg\""
Jul 13 22:01:47.712762 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.712748    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 22:01:47.720682 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.720660    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"]
Jul 13 22:01:47.737806 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.737784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:47.793143 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.793116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxtdm\" (UniqueName: \"kubernetes.io/projected/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-kube-api-access-sxtdm\") pod \"maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:47.793257 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.793169    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-openshift-service-ca\") pod \"maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:47.793304 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.793266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-maas-api-tls\") pod \"maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:47.840243 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.840207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:47.894056 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.893984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxtdm\" (UniqueName: \"kubernetes.io/projected/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-kube-api-access-sxtdm\") pod \"maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:47.894056 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.894020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-openshift-service-ca\") pod \"maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:47.894274 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.894183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-maas-api-tls\") pod \"maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:47.894346 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:01:47.894328    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-serving-cert: secret "maas-api-e2e-derive-628bb6aa-serving-cert" not found
Jul 13 22:01:47.894404 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:01:47.894393    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-maas-api-tls podName:55a8cf5f-e19c-4ded-88f5-4232a8bb4891 nodeName:}" failed. No retries permitted until 2026-07-13 22:01:48.39437263 +0000 UTC m=+1949.550766450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-maas-api-tls") pod "maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp" (UID: "55a8cf5f-e19c-4ded-88f5-4232a8bb4891") : secret "maas-api-e2e-derive-628bb6aa-serving-cert" not found
Jul 13 22:01:47.894701 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.894685    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-openshift-service-ca\") pod \"maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:47.906995 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:47.906969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxtdm\" (UniqueName: \"kubernetes.io/projected/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-kube-api-access-sxtdm\") pod \"maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:48.095454 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:48.095411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:48.288116 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:48.288078    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:48.389334 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:48.389307    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:48.397373 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:48.397351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-maas-api-tls\") pod \"maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:48.399668 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:48.399652    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-maas-api-tls\") pod \"maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:48.620959 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:48.620879    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:48.754832 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:48.754797    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"]
Jul 13 22:01:48.758309 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:01:48.758276    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a8cf5f_e19c_4ded_88f5_4232a8bb4891.slice/crio-1f22a1ddfcfe75c4a49c018d6d47ec0b364831d9a88a29dd129bf7138bdd4399 WatchSource:0}: Error finding container 1f22a1ddfcfe75c4a49c018d6d47ec0b364831d9a88a29dd129bf7138bdd4399: Status 404 returned error can't find the container with id 1f22a1ddfcfe75c4a49c018d6d47ec0b364831d9a88a29dd129bf7138bdd4399
Jul 13 22:01:49.498735 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:49.498701    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp" event={"ID":"55a8cf5f-e19c-4ded-88f5-4232a8bb4891","Type":"ContainerStarted","Data":"ba6f8281003bce6548b59ccbeb63b101cd252e6ad1eb3663b69a08d789da6555"}
Jul 13 22:01:49.498872 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:49.498756    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp" event={"ID":"55a8cf5f-e19c-4ded-88f5-4232a8bb4891","Type":"ContainerStarted","Data":"1f22a1ddfcfe75c4a49c018d6d47ec0b364831d9a88a29dd129bf7138bdd4399"}
Jul 13 22:01:49.498872 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:49.498777    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:49.516681 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:49.516640    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp" podStartSLOduration=2.059669752 podStartE2EDuration="2.51662314s" totalImagesPullingTime="456.953388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:49.515723702 +0000 UTC m=+1950.672117543" watchObservedRunningTime="2026-07-13 22:01:49.51662314 +0000 UTC m=+1950.673016984"
Jul 13 22:01:51.938374 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:51.938339    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"]
Jul 13 22:01:51.941560 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:51.941537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"
Jul 13 22:01:51.944127 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:51.944107    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:01:51.950817 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:51.950792    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"]
Jul 13 22:01:52.023772 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:52.023739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aff189a-d037-4375-a6bd-5553576b1dd8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf\" (UID: \"8aff189a-d037-4375-a6bd-5553576b1dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"
Jul 13 22:01:52.124445 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:52.124407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aff189a-d037-4375-a6bd-5553576b1dd8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf\" (UID: \"8aff189a-d037-4375-a6bd-5553576b1dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"
Jul 13 22:01:52.125026 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:52.125006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aff189a-d037-4375-a6bd-5553576b1dd8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf\" (UID: \"8aff189a-d037-4375-a6bd-5553576b1dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"
Jul 13 22:01:52.253492 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:52.253456    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"
Jul 13 22:01:52.385792 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:52.385764    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"]
Jul 13 22:01:52.387403 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:01:52.387376    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aff189a_d037_4375_a6bd_5553576b1dd8.slice/crio-489dcc30d5298930086c274e6d8fca2c1ba658dca0ee18eeeda302b22ad14a22 WatchSource:0}: Error finding container 489dcc30d5298930086c274e6d8fca2c1ba658dca0ee18eeeda302b22ad14a22: Status 404 returned error can't find the container with id 489dcc30d5298930086c274e6d8fca2c1ba658dca0ee18eeeda302b22ad14a22
Jul 13 22:01:52.483206 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:52.483139    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:52.514590 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:52.514507    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf" event={"ID":"8aff189a-d037-4375-a6bd-5553576b1dd8","Type":"ContainerStarted","Data":"489dcc30d5298930086c274e6d8fca2c1ba658dca0ee18eeeda302b22ad14a22"}
Jul 13 22:01:52.582861 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:52.582827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:01:53.522045 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:53.522011    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerID="c28a9624d3609e10f3f14cb23e8344726119a5ae609b9b864d101436eb82d4df" exitCode=7
Jul 13 22:01:53.522584 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:53.522097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf" event={"ID":"8aff189a-d037-4375-a6bd-5553576b1dd8","Type":"ContainerDied","Data":"c28a9624d3609e10f3f14cb23e8344726119a5ae609b9b864d101436eb82d4df"}
Jul 13 22:01:53.522584 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:53.522337    2582 scope.go:117] "RemoveContainer" containerID="c28a9624d3609e10f3f14cb23e8344726119a5ae609b9b864d101436eb82d4df"
Jul 13 22:01:54.528221 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:54.528189    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerID="20a707a310b4dd51d341fcd11f8ae43e4c2cc1eb2e058c257c3452647c4fc068" exitCode=7
Jul 13 22:01:54.528638 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:54.528285    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf" event={"ID":"8aff189a-d037-4375-a6bd-5553576b1dd8","Type":"ContainerDied","Data":"20a707a310b4dd51d341fcd11f8ae43e4c2cc1eb2e058c257c3452647c4fc068"}
Jul 13 22:01:54.528638 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:54.528344    2582 scope.go:117] "RemoveContainer" containerID="c28a9624d3609e10f3f14cb23e8344726119a5ae609b9b864d101436eb82d4df"
Jul 13 22:01:54.528638 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:54.528474    2582 scope.go:117] "RemoveContainer" containerID="20a707a310b4dd51d341fcd11f8ae43e4c2cc1eb2e058c257c3452647c4fc068"
Jul 13 22:01:54.528779 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:01:54.528724    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf_odh-ai-gateway-infra(8aff189a-d037-4375-a6bd-5553576b1dd8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8"
Jul 13 22:01:54.758348 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:54.758312    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:01:55.509828 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:55.509801    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:01:55.534891 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:55.534867    2582 scope.go:117] "RemoveContainer" containerID="20a707a310b4dd51d341fcd11f8ae43e4c2cc1eb2e058c257c3452647c4fc068"
Jul 13 22:01:55.535347 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:01:55.535308    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf_odh-ai-gateway-infra(8aff189a-d037-4375-a6bd-5553576b1dd8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8"
Jul 13 22:01:55.880231 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:01:55.880197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:02:02.739949 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:02.739911    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:02:06.358736 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:06.358706    2582 scope.go:117] "RemoveContainer" containerID="20a707a310b4dd51d341fcd11f8ae43e4c2cc1eb2e058c257c3452647c4fc068"
Jul 13 22:02:07.593672 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:07.593635    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerID="17a560744aa5f7afdf6a2e425d7b4ad7091c2c404cae6e80cf4dbdfae01ece4d" exitCode=0
Jul 13 22:02:07.594106 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:07.593686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf" event={"ID":"8aff189a-d037-4375-a6bd-5553576b1dd8","Type":"ContainerDied","Data":"17a560744aa5f7afdf6a2e425d7b4ad7091c2c404cae6e80cf4dbdfae01ece4d"}
Jul 13 22:02:07.594106 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:07.593746    2582 scope.go:117] "RemoveContainer" containerID="20a707a310b4dd51d341fcd11f8ae43e4c2cc1eb2e058c257c3452647c4fc068"
Jul 13 22:02:08.627533 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:08.627498    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"]
Jul 13 22:02:08.731816 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:08.731793    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"
Jul 13 22:02:08.767019 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:08.766996    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aff189a-d037-4375-a6bd-5553576b1dd8-maas-api-service-ca\") pod \"8aff189a-d037-4375-a6bd-5553576b1dd8\" (UID: \"8aff189a-d037-4375-a6bd-5553576b1dd8\") "
Jul 13 22:02:08.767365 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:08.767345    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aff189a-d037-4375-a6bd-5553576b1dd8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8aff189a-d037-4375-a6bd-5553576b1dd8" (UID: "8aff189a-d037-4375-a6bd-5553576b1dd8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:02:08.867620 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:08.867547    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aff189a-d037-4375-a6bd-5553576b1dd8-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:02:09.605717 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:09.605692    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"
Jul 13 22:02:09.605879 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:09.605719    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf" event={"ID":"8aff189a-d037-4375-a6bd-5553576b1dd8","Type":"ContainerDied","Data":"489dcc30d5298930086c274e6d8fca2c1ba658dca0ee18eeeda302b22ad14a22"}
Jul 13 22:02:09.605879 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:09.605768    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489dcc30d5298930086c274e6d8fca2c1ba658dca0ee18eeeda302b22ad14a22"
Jul 13 22:02:09.638214 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:09.638136    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"]
Jul 13 22:02:09.641531 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:09.641511    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-lnpcf"]
Jul 13 22:02:11.363840 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:11.363743    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8" path="/var/lib/kubelet/pods/8aff189a-d037-4375-a6bd-5553576b1dd8/volumes"
Jul 13 22:02:41.982928 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.982884    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8"]
Jul 13 22:02:41.983502 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.983480    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerName="revoke-keys"
Jul 13 22:02:41.983547 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.983506    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerName="revoke-keys"
Jul 13 22:02:41.983547 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.983527    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerName="revoke-keys"
Jul 13 22:02:41.983547 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.983536    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerName="revoke-keys"
Jul 13 22:02:41.983685 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.983547    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerName="revoke-keys"
Jul 13 22:02:41.983685 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.983557    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerName="revoke-keys"
Jul 13 22:02:41.983685 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.983650    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerName="revoke-keys"
Jul 13 22:02:41.983685 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.983662    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerName="revoke-keys"
Jul 13 22:02:41.987129 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.987107    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8"
Jul 13 22:02:41.990090 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.990069    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:02:41.995050 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:41.995028    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8"]
Jul 13 22:02:42.159281 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:42.159239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8\" (UID: \"c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8"
Jul 13 22:02:42.260612 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:42.260523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8\" (UID: \"c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8"
Jul 13 22:02:42.261201 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:42.261145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8\" (UID: \"c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8"
Jul 13 22:02:42.299387 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:42.299362    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8"
Jul 13 22:02:42.634071 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:42.633995    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8"]
Jul 13 22:02:42.637321 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:02:42.637292    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9358e6a_f5a5_4ae5_9f45_40f6014b6ee1.slice/crio-6f51aeae06fa24d23082e18b3fa3a17733c1ac0b88f8f2da5791fa4cf3b9fd23 WatchSource:0}: Error finding container 6f51aeae06fa24d23082e18b3fa3a17733c1ac0b88f8f2da5791fa4cf3b9fd23: Status 404 returned error can't find the container with id 6f51aeae06fa24d23082e18b3fa3a17733c1ac0b88f8f2da5791fa4cf3b9fd23
Jul 13 22:02:42.765337 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:42.765304    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8" event={"ID":"c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1","Type":"ContainerStarted","Data":"6f51aeae06fa24d23082e18b3fa3a17733c1ac0b88f8f2da5791fa4cf3b9fd23"}
Jul 13 22:02:43.772330 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:43.772293    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1" containerID="c392d5955d3967fe906c10f78eaec997b2d4252e4ceb2c1e4b8908de60ca458c" exitCode=0
Jul 13 22:02:43.772719 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:43.772405    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8" event={"ID":"c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1","Type":"ContainerDied","Data":"c392d5955d3967fe906c10f78eaec997b2d4252e4ceb2c1e4b8908de60ca458c"}
Jul 13 22:02:44.914876 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:44.914853    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8"
Jul 13 22:02:44.979594 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:44.979563    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1-maas-api-service-ca\") pod \"c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1\" (UID: \"c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1\") "
Jul 13 22:02:44.979924 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:44.979899    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1" (UID: "c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:02:45.080385 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:45.080305    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:02:45.784214 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:45.784177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8" event={"ID":"c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1","Type":"ContainerDied","Data":"6f51aeae06fa24d23082e18b3fa3a17733c1ac0b88f8f2da5791fa4cf3b9fd23"}
Jul 13 22:02:45.784214 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:45.784204    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8"
Jul 13 22:02:45.784411 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:45.784237    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f51aeae06fa24d23082e18b3fa3a17733c1ac0b88f8f2da5791fa4cf3b9fd23"
Jul 13 22:02:52.655884 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.655851    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"]
Jul 13 22:02:52.656356 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.656106    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp" podUID="55a8cf5f-e19c-4ded-88f5-4232a8bb4891" containerName="maas-api" containerID="cri-o://ba6f8281003bce6548b59ccbeb63b101cd252e6ad1eb3663b69a08d789da6555" gracePeriod=30
Jul 13 22:02:52.685143 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.685116    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:02:52.818103 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.818065    2582 generic.go:356] "Generic (PLEG): container finished" podID="55a8cf5f-e19c-4ded-88f5-4232a8bb4891" containerID="ba6f8281003bce6548b59ccbeb63b101cd252e6ad1eb3663b69a08d789da6555" exitCode=0
Jul 13 22:02:52.818280 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.818139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp" event={"ID":"55a8cf5f-e19c-4ded-88f5-4232a8bb4891","Type":"ContainerDied","Data":"ba6f8281003bce6548b59ccbeb63b101cd252e6ad1eb3663b69a08d789da6555"}
Jul 13 22:02:52.906379 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.906319    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:02:52.935638 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.935609    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-openshift-service-ca\") pod \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") "
Jul 13 22:02:52.935775 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.935676    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxtdm\" (UniqueName: \"kubernetes.io/projected/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-kube-api-access-sxtdm\") pod \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") "
Jul 13 22:02:52.935775 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.935696    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-maas-api-tls\") pod \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\" (UID: \"55a8cf5f-e19c-4ded-88f5-4232a8bb4891\") "
Jul 13 22:02:52.935950 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.935928    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55a8cf5f-e19c-4ded-88f5-4232a8bb4891" (UID: "55a8cf5f-e19c-4ded-88f5-4232a8bb4891"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:02:52.937693 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.937662    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55a8cf5f-e19c-4ded-88f5-4232a8bb4891" (UID: "55a8cf5f-e19c-4ded-88f5-4232a8bb4891"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:02:52.937788 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:52.937719    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-kube-api-access-sxtdm" (OuterVolumeSpecName: "kube-api-access-sxtdm") pod "55a8cf5f-e19c-4ded-88f5-4232a8bb4891" (UID: "55a8cf5f-e19c-4ded-88f5-4232a8bb4891"). InnerVolumeSpecName "kube-api-access-sxtdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:02:53.036992 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:53.036960    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:02:53.036992 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:53.036989    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxtdm\" (UniqueName: \"kubernetes.io/projected/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-kube-api-access-sxtdm\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:02:53.037140 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:53.037003    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55a8cf5f-e19c-4ded-88f5-4232a8bb4891-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:02:53.824215 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:53.824185    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"
Jul 13 22:02:53.824611 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:53.824185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp" event={"ID":"55a8cf5f-e19c-4ded-88f5-4232a8bb4891","Type":"ContainerDied","Data":"1f22a1ddfcfe75c4a49c018d6d47ec0b364831d9a88a29dd129bf7138bdd4399"}
Jul 13 22:02:53.824611 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:53.824339    2582 scope.go:117] "RemoveContainer" containerID="ba6f8281003bce6548b59ccbeb63b101cd252e6ad1eb3663b69a08d789da6555"
Jul 13 22:02:53.841839 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:53.841813    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"]
Jul 13 22:02:53.845696 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:53.845669    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-628bb6aa-6c4fcfd57c-65vlp"]
Jul 13 22:02:55.363756 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:55.363720    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55a8cf5f-e19c-4ded-88f5-4232a8bb4891" path="/var/lib/kubelet/pods/55a8cf5f-e19c-4ded-88f5-4232a8bb4891/volumes"
Jul 13 22:02:55.958829 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:55.958798    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:02:57.084421 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:02:57.084377    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:03.111691 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:03.111630    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:03.213676 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:03.213639    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:07.280681 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:07.280627    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:07.916837 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:07.916804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:08.016436 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:08.016388    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:08.117568 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:08.117520    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:09.594795 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:09.594752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:12.398052 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:12.398018    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:13.618613 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:13.618573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:14.873454 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:14.873413    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:15.851863 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:15.851829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:15.960174 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:15.960123    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:16.053462 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:16.053427    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:16.153269 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:16.153177    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:16.704816 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:16.704771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:42.626193 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:42.626091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:48.841787 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:48.841743    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:48.919752 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:48.919713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:49.054467 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:49.054433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:03:49.143677 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:03:49.143589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:04.991466 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:04.991428    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:10.875394 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:10.875361    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:11.007223 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:11.007183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:11.099501 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:11.099468    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:19.359578 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:19.359549    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 22:04:19.364104 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:19.364079    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 22:04:19.365172 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:19.365133    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 22:04:19.368843 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:19.368820    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 22:04:19.431642 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:19.431620    2582 scope.go:117] "RemoveContainer" containerID="474d7fb24904d34cde43045d0091c21534559a077aaa662e6ebcbbb72a48fdb6"
Jul 13 22:04:21.601312 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:21.601271    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:22.416530 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:22.416497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:28.372767 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:28.372732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:28.440544 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:28.440515    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:28.550883 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:28.550851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:28.649255 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:28.649142    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:28.785968 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:28.785935    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:29.562131 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:29.562090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:29.662114 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:29.662085    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:29.770168 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:29.770120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:31.469633 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:31.469599    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:32.063333 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:32.063296    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:04:37.557850 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:37.557818    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:37.654401 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:37.654358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:38.247487 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:38.247448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:41.260966 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:41.260929    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:04:44.570528 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:44.570490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:44.672538 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:44.672477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:44.813581 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:44.813552    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:45.137519 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:45.137480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:45.757940 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:45.757900    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:47.927905 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:47.927869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:48.015107 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.015074    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"]
Jul 13 22:04:48.015468 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.015451    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55a8cf5f-e19c-4ded-88f5-4232a8bb4891" containerName="maas-api"
Jul 13 22:04:48.015468 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.015468    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="55a8cf5f-e19c-4ded-88f5-4232a8bb4891" containerName="maas-api"
Jul 13 22:04:48.015595 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.015490    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1" containerName="revoke-keys"
Jul 13 22:04:48.015595 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.015497    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1" containerName="revoke-keys"
Jul 13 22:04:48.015595 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.015553    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1" containerName="revoke-keys"
Jul 13 22:04:48.015595 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.015566    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aff189a-d037-4375-a6bd-5553576b1dd8" containerName="revoke-keys"
Jul 13 22:04:48.015595 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.015574    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="55a8cf5f-e19c-4ded-88f5-4232a8bb4891" containerName="maas-api"
Jul 13 22:04:48.018287 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.018265    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.019194 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.019139    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:48.021708 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.021690    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 22:04:48.021878 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.021860    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a941a48d-serving-cert\""
Jul 13 22:04:48.022275 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.022255    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 22:04:48.022749 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.022736    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kfxsg\""
Jul 13 22:04:48.031205 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.031183    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"]
Jul 13 22:04:48.099179 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.097012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:48.107803 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.107776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85mlc\" (UniqueName: \"kubernetes.io/projected/19bb9950-48c5-4509-8a16-08f16b5c87fc-kube-api-access-85mlc\") pod \"maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.107956 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.107835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19bb9950-48c5-4509-8a16-08f16b5c87fc-openshift-service-ca\") pod \"maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.107956 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.107925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bb9950-48c5-4509-8a16-08f16b5c87fc-maas-api-tls\") pod \"maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.209213 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.209090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19bb9950-48c5-4509-8a16-08f16b5c87fc-openshift-service-ca\") pod \"maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.209213 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.209184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bb9950-48c5-4509-8a16-08f16b5c87fc-maas-api-tls\") pod \"maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.209424 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.209233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85mlc\" (UniqueName: \"kubernetes.io/projected/19bb9950-48c5-4509-8a16-08f16b5c87fc-kube-api-access-85mlc\") pod \"maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.209424 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:04:48.209339    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-serving-cert: secret "maas-api-e2e-mt-a941a48d-serving-cert" not found
Jul 13 22:04:48.209424 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:04:48.209418    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bb9950-48c5-4509-8a16-08f16b5c87fc-maas-api-tls podName:19bb9950-48c5-4509-8a16-08f16b5c87fc nodeName:}" failed. No retries permitted until 2026-07-13 22:04:48.709397531 +0000 UTC m=+2129.865791354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19bb9950-48c5-4509-8a16-08f16b5c87fc-maas-api-tls") pod "maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8" (UID: "19bb9950-48c5-4509-8a16-08f16b5c87fc") : secret "maas-api-e2e-mt-a941a48d-serving-cert" not found
Jul 13 22:04:48.209771 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.209750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19bb9950-48c5-4509-8a16-08f16b5c87fc-openshift-service-ca\") pod \"maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.224235 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.224212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85mlc\" (UniqueName: \"kubernetes.io/projected/19bb9950-48c5-4509-8a16-08f16b5c87fc-kube-api-access-85mlc\") pod \"maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.514507 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.514476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:48.712723 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.712692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bb9950-48c5-4509-8a16-08f16b5c87fc-maas-api-tls\") pod \"maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.715212 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.715185    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bb9950-48c5-4509-8a16-08f16b5c87fc-maas-api-tls\") pod \"maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.853770 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.853685    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:48.928833 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.928803    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:48.953920 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:48.953895    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:49.064904 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:49.064873    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"]
Jul 13 22:04:49.066245 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:04:49.066215    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bb9950_48c5_4509_8a16_08f16b5c87fc.slice/crio-e97fdb73120b74220b6758373a62a10598f6a953479d4fed0662c3f5ad9ff1dd WatchSource:0}: Error finding container e97fdb73120b74220b6758373a62a10598f6a953479d4fed0662c3f5ad9ff1dd: Status 404 returned error can't find the container with id e97fdb73120b74220b6758373a62a10598f6a953479d4fed0662c3f5ad9ff1dd
Jul 13 22:04:49.067645 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:49.067625    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 22:04:49.371864 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:49.371829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8" event={"ID":"19bb9950-48c5-4509-8a16-08f16b5c87fc","Type":"ContainerStarted","Data":"e97fdb73120b74220b6758373a62a10598f6a953479d4fed0662c3f5ad9ff1dd"}
Jul 13 22:04:50.378564 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:50.378524    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8" event={"ID":"19bb9950-48c5-4509-8a16-08f16b5c87fc","Type":"ContainerStarted","Data":"a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b"}
Jul 13 22:04:50.378948 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:50.378582    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:50.399584 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:50.399533    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8" podStartSLOduration=2.932675742 podStartE2EDuration="3.399515765s" totalImagesPullingTime="466.840023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:04:50.398953833 +0000 UTC m=+2131.555347675" watchObservedRunningTime="2026-07-13 22:04:50.399515765 +0000 UTC m=+2131.555909608"
Jul 13 22:04:52.451953 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:52.451924    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:53.105756 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:53.105720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:53.204425 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:53.204380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:53.467087 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:53.467002    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:04:53.574655 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:53.574623    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:56.389058 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:56.389021    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:04:59.832076 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:59.832044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:04:59.932841 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:04:59.932810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:00.210959 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:00.210881    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4"]
Jul 13 22:05:00.214452 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:00.214428    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4"
Jul 13 22:05:00.220702 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:00.220463    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:05:00.228271 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:00.228249    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4"]
Jul 13 22:05:00.315463 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:00.315431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2607e304-03ae-480f-bfe0-57a15a8c204b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a941a48d-7msx4\" (UID: \"2607e304-03ae-480f-bfe0-57a15a8c204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4"
Jul 13 22:05:00.416646 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:00.416618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2607e304-03ae-480f-bfe0-57a15a8c204b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a941a48d-7msx4\" (UID: \"2607e304-03ae-480f-bfe0-57a15a8c204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4"
Jul 13 22:05:00.417246 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:00.417228    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2607e304-03ae-480f-bfe0-57a15a8c204b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a941a48d-7msx4\" (UID: \"2607e304-03ae-480f-bfe0-57a15a8c204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4"
Jul 13 22:05:00.523675 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:00.523654    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4"
Jul 13 22:05:00.656256 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:00.656230    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4"]
Jul 13 22:05:00.659211 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:05:00.659184    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2607e304_03ae_480f_bfe0_57a15a8c204b.slice/crio-68dfa07df5f7c9c9d9fdab9a290785f81fe14dea6300df45b0073ba66d4ab17a WatchSource:0}: Error finding container 68dfa07df5f7c9c9d9fdab9a290785f81fe14dea6300df45b0073ba66d4ab17a: Status 404 returned error can't find the container with id 68dfa07df5f7c9c9d9fdab9a290785f81fe14dea6300df45b0073ba66d4ab17a
Jul 13 22:05:01.429681 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:01.429637    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4" event={"ID":"2607e304-03ae-480f-bfe0-57a15a8c204b","Type":"ContainerStarted","Data":"c64fe0b2bf97754ed666547f2d6ac2901c137af5090898bfaa628285dd15dd8f"}
Jul 13 22:05:01.430092 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:01.429717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4" event={"ID":"2607e304-03ae-480f-bfe0-57a15a8c204b","Type":"ContainerStarted","Data":"68dfa07df5f7c9c9d9fdab9a290785f81fe14dea6300df45b0073ba66d4ab17a"}
Jul 13 22:05:01.446862 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:01.446813    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4" podStartSLOduration=1.4467967179999999 podStartE2EDuration="1.446796718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:01.445408842 +0000 UTC m=+2142.601802677" watchObservedRunningTime="2026-07-13 22:05:01.446796718 +0000 UTC m=+2142.603190558"
Jul 13 22:05:02.435956 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:02.435919    2582 generic.go:356] "Generic (PLEG): container finished" podID="2607e304-03ae-480f-bfe0-57a15a8c204b" containerID="c64fe0b2bf97754ed666547f2d6ac2901c137af5090898bfaa628285dd15dd8f" exitCode=0
Jul 13 22:05:02.436356 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:02.436004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4" event={"ID":"2607e304-03ae-480f-bfe0-57a15a8c204b","Type":"ContainerDied","Data":"c64fe0b2bf97754ed666547f2d6ac2901c137af5090898bfaa628285dd15dd8f"}
Jul 13 22:05:03.567462 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:03.567438    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4"
Jul 13 22:05:03.744676 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:03.744645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2607e304-03ae-480f-bfe0-57a15a8c204b-maas-api-service-ca\") pod \"2607e304-03ae-480f-bfe0-57a15a8c204b\" (UID: \"2607e304-03ae-480f-bfe0-57a15a8c204b\") "
Jul 13 22:05:03.745059 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:03.745038    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2607e304-03ae-480f-bfe0-57a15a8c204b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2607e304-03ae-480f-bfe0-57a15a8c204b" (UID: "2607e304-03ae-480f-bfe0-57a15a8c204b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:05:03.845973 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:03.845930    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2607e304-03ae-480f-bfe0-57a15a8c204b-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:05:04.448176 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:04.448127    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4"
Jul 13 22:05:04.448176 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:04.448169    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a941a48d-7msx4" event={"ID":"2607e304-03ae-480f-bfe0-57a15a8c204b","Type":"ContainerDied","Data":"68dfa07df5f7c9c9d9fdab9a290785f81fe14dea6300df45b0073ba66d4ab17a"}
Jul 13 22:05:04.448386 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:04.448216    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68dfa07df5f7c9c9d9fdab9a290785f81fe14dea6300df45b0073ba66d4ab17a"
Jul 13 22:05:10.701915 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:10.701885    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"]
Jul 13 22:05:10.702451 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:10.702100    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8" podUID="19bb9950-48c5-4509-8a16-08f16b5c87fc" containerName="maas-api" containerID="cri-o://a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b" gracePeriod=30
Jul 13 22:05:10.729861 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:10.729827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:10.951714 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:10.951692    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:05:11.109054 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.109015    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85mlc\" (UniqueName: \"kubernetes.io/projected/19bb9950-48c5-4509-8a16-08f16b5c87fc-kube-api-access-85mlc\") pod \"19bb9950-48c5-4509-8a16-08f16b5c87fc\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") "
Jul 13 22:05:11.109225 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.109072    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19bb9950-48c5-4509-8a16-08f16b5c87fc-openshift-service-ca\") pod \"19bb9950-48c5-4509-8a16-08f16b5c87fc\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") "
Jul 13 22:05:11.109225 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.109201    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bb9950-48c5-4509-8a16-08f16b5c87fc-maas-api-tls\") pod \"19bb9950-48c5-4509-8a16-08f16b5c87fc\" (UID: \"19bb9950-48c5-4509-8a16-08f16b5c87fc\") "
Jul 13 22:05:11.109402 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.109385    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bb9950-48c5-4509-8a16-08f16b5c87fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19bb9950-48c5-4509-8a16-08f16b5c87fc" (UID: "19bb9950-48c5-4509-8a16-08f16b5c87fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:05:11.109470 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.109457    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19bb9950-48c5-4509-8a16-08f16b5c87fc-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:05:11.111116 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.111091    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19bb9950-48c5-4509-8a16-08f16b5c87fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19bb9950-48c5-4509-8a16-08f16b5c87fc" (UID: "19bb9950-48c5-4509-8a16-08f16b5c87fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:05:11.111218 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.111182    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bb9950-48c5-4509-8a16-08f16b5c87fc-kube-api-access-85mlc" (OuterVolumeSpecName: "kube-api-access-85mlc") pod "19bb9950-48c5-4509-8a16-08f16b5c87fc" (UID: "19bb9950-48c5-4509-8a16-08f16b5c87fc"). InnerVolumeSpecName "kube-api-access-85mlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:05:11.209939 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.209906    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85mlc\" (UniqueName: \"kubernetes.io/projected/19bb9950-48c5-4509-8a16-08f16b5c87fc-kube-api-access-85mlc\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:05:11.209939 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.209937    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bb9950-48c5-4509-8a16-08f16b5c87fc-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:05:11.484533 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.484506    2582 generic.go:356] "Generic (PLEG): container finished" podID="19bb9950-48c5-4509-8a16-08f16b5c87fc" containerID="a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b" exitCode=0
Jul 13 22:05:11.484674 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.484552    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"
Jul 13 22:05:11.484674 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.484554    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8" event={"ID":"19bb9950-48c5-4509-8a16-08f16b5c87fc","Type":"ContainerDied","Data":"a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b"}
Jul 13 22:05:11.484674 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.484668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8" event={"ID":"19bb9950-48c5-4509-8a16-08f16b5c87fc","Type":"ContainerDied","Data":"e97fdb73120b74220b6758373a62a10598f6a953479d4fed0662c3f5ad9ff1dd"}
Jul 13 22:05:11.484848 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.484690    2582 scope.go:117] "RemoveContainer" containerID="a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b"
Jul 13 22:05:11.494901 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.494875    2582 scope.go:117] "RemoveContainer" containerID="a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b"
Jul 13 22:05:11.495306 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:05:11.495276    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b\": container with ID starting with a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b not found: ID does not exist" containerID="a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b"
Jul 13 22:05:11.495403 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.495319    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b"} err="failed to get container status \"a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b\": rpc error: code = NotFound desc = could not find container \"a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b\": container with ID starting with a43ba281a8cd16ea8378ac33f8a8c8df5594e97808f34e2b969d5053b81be87b not found: ID does not exist"
Jul 13 22:05:11.508893 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.508872    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"]
Jul 13 22:05:11.514960 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:11.514938    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a941a48d-8c6c46d75-rdjb8"]
Jul 13 22:05:13.363323 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:13.363286    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19bb9950-48c5-4509-8a16-08f16b5c87fc" path="/var/lib/kubelet/pods/19bb9950-48c5-4509-8a16-08f16b5c87fc/volumes"
Jul 13 22:05:13.663243 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:13.663163    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:05:15.354265 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:15.354225    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:21.427475 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:21.427437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:21.523104 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:21.523072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:26.036321 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:26.036288    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:26.254188 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:26.254140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:26.407969 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:26.407889    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:26.498239 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:26.498210    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:26.603431 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:26.603392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:32.604504 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:32.604461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:32.710122 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:32.710092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:32.801757 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:32.801725    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:32.962792 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:32.962704    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:38.938373 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:38.938339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:39.041804 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:39.041761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:45.824496 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:45.824462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:46.167380 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:46.167293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:46.510866 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:46.510836    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:49.152393 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.152351    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:49.248475 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.248441    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"]
Jul 13 22:05:49.248829 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.248807    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bb9950-48c5-4509-8a16-08f16b5c87fc" containerName="maas-api"
Jul 13 22:05:49.248963 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.248837    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bb9950-48c5-4509-8a16-08f16b5c87fc" containerName="maas-api"
Jul 13 22:05:49.248963 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.248848    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2607e304-03ae-480f-bfe0-57a15a8c204b" containerName="revoke-keys"
Jul 13 22:05:49.248963 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.248854    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2607e304-03ae-480f-bfe0-57a15a8c204b" containerName="revoke-keys"
Jul 13 22:05:49.248963 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.248924    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bb9950-48c5-4509-8a16-08f16b5c87fc" containerName="maas-api"
Jul 13 22:05:49.248963 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.248937    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2607e304-03ae-480f-bfe0-57a15a8c204b" containerName="revoke-keys"
Jul 13 22:05:49.252535 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.252518    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.255533 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.255502    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-06fcd095-serving-cert\""
Jul 13 22:05:49.255750 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.255737    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kfxsg\""
Jul 13 22:05:49.256808 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.256791    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 22:05:49.256902 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.256796    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 22:05:49.264935 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.264913    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"]
Jul 13 22:05:49.268458 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.268421    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:49.310768 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.310736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njqj7\" (UniqueName: \"kubernetes.io/projected/e678cff5-7fd4-4303-9753-d3a36266d7cf-kube-api-access-njqj7\") pod \"maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.310911 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.310810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678cff5-7fd4-4303-9753-d3a36266d7cf-maas-api-tls\") pod \"maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.310911 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.310842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e678cff5-7fd4-4303-9753-d3a36266d7cf-openshift-service-ca\") pod \"maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.360593 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.360564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:49.411829 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.411739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqj7\" (UniqueName: \"kubernetes.io/projected/e678cff5-7fd4-4303-9753-d3a36266d7cf-kube-api-access-njqj7\") pod \"maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.411829 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.411819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678cff5-7fd4-4303-9753-d3a36266d7cf-maas-api-tls\") pod \"maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.412043 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.411852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e678cff5-7fd4-4303-9753-d3a36266d7cf-openshift-service-ca\") pod \"maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.412043 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:05:49.411955    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-serving-cert: secret "maas-api-e2e-mt-06fcd095-serving-cert" not found
Jul 13 22:05:49.412043 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:05:49.412036    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e678cff5-7fd4-4303-9753-d3a36266d7cf-maas-api-tls podName:e678cff5-7fd4-4303-9753-d3a36266d7cf nodeName:}" failed. No retries permitted until 2026-07-13 22:05:49.912016156 +0000 UTC m=+2191.068409985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e678cff5-7fd4-4303-9753-d3a36266d7cf-maas-api-tls") pod "maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr" (UID: "e678cff5-7fd4-4303-9753-d3a36266d7cf") : secret "maas-api-e2e-mt-06fcd095-serving-cert" not found
Jul 13 22:05:49.412570 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.412549    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e678cff5-7fd4-4303-9753-d3a36266d7cf-openshift-service-ca\") pod \"maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.429186 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.429163    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqj7\" (UniqueName: \"kubernetes.io/projected/e678cff5-7fd4-4303-9753-d3a36266d7cf-kube-api-access-njqj7\") pod \"maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.685771 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.685681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:49.859529 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.859495    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:49.916089 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.916058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678cff5-7fd4-4303-9753-d3a36266d7cf-maas-api-tls\") pod \"maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.918295 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.918266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678cff5-7fd4-4303-9753-d3a36266d7cf-maas-api-tls\") pod \"maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:49.957539 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:49.957466    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:50.164175 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:50.164114    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:50.297163 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:50.297110    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"]
Jul 13 22:05:50.299771 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:05:50.299734    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode678cff5_7fd4_4303_9753_d3a36266d7cf.slice/crio-e9cecbc0a4a3a1faab1f8afa8f2eef7de04c4727fffc804a10084ed2dd5994b2 WatchSource:0}: Error finding container e9cecbc0a4a3a1faab1f8afa8f2eef7de04c4727fffc804a10084ed2dd5994b2: Status 404 returned error can't find the container with id e9cecbc0a4a3a1faab1f8afa8f2eef7de04c4727fffc804a10084ed2dd5994b2
Jul 13 22:05:50.669253 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:50.669218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr" event={"ID":"e678cff5-7fd4-4303-9753-d3a36266d7cf","Type":"ContainerStarted","Data":"e9cecbc0a4a3a1faab1f8afa8f2eef7de04c4727fffc804a10084ed2dd5994b2"}
Jul 13 22:05:51.675512 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:51.675475    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr" event={"ID":"e678cff5-7fd4-4303-9753-d3a36266d7cf","Type":"ContainerStarted","Data":"de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036"}
Jul 13 22:05:51.675888 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:51.675539    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:05:51.696566 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:51.696518    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr" podStartSLOduration=2.257978866 podStartE2EDuration="2.696502696s" totalImagesPullingTime="438.52383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:51.6942065 +0000 UTC m=+2192.850600343" watchObservedRunningTime="2026-07-13 22:05:51.696502696 +0000 UTC m=+2192.852896539"
Jul 13 22:05:53.626824 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:53.626788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:05:54.257664 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:54.257631    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:05:57.686560 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:05:57.686531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:06:02.871205 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:02.871167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:06:04.212573 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:04.212522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:06:04.313123 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:04.313086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:06:06.068020 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:06.067985    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:06:10.818831 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:10.818775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:06:10.919245 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:10.919213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:06:18.458112 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:18.458077    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:06:19.460739 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:19.460708    2582 scope.go:117] "RemoveContainer" containerID="2940fc7aa5f8be7f25e09c9d33277c8823f97f557dcdbe68a946025b02083d19"
Jul 13 22:06:19.468616 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:19.468596    2582 scope.go:117] "RemoveContainer" containerID="4b43a623d28d8975409c85b5883b5a5c4b8b1e32e01e4bf26e400110cd7fac56"
Jul 13 22:06:21.698441 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:21.698407    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx"]
Jul 13 22:06:21.703648 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:21.703626    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx"
Jul 13 22:06:21.706317 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:21.706292    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:06:21.710594 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:21.710545    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx"]
Jul 13 22:06:21.800281 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:21.800253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0023c91c-ea2f-4f3d-ae34-d31c4a8a842b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-06fcd095-vzggx\" (UID: \"0023c91c-ea2f-4f3d-ae34-d31c4a8a842b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx"
Jul 13 22:06:21.901350 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:21.901319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0023c91c-ea2f-4f3d-ae34-d31c4a8a842b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-06fcd095-vzggx\" (UID: \"0023c91c-ea2f-4f3d-ae34-d31c4a8a842b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx"
Jul 13 22:06:21.901952 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:21.901931    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0023c91c-ea2f-4f3d-ae34-d31c4a8a842b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-06fcd095-vzggx\" (UID: \"0023c91c-ea2f-4f3d-ae34-d31c4a8a842b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx"
Jul 13 22:06:22.016182 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:22.016129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx"
Jul 13 22:06:22.360417 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:22.360344    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx"]
Jul 13 22:06:22.362643 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:06:22.362619    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0023c91c_ea2f_4f3d_ae34_d31c4a8a842b.slice/crio-ad11e55e213227150957f4065088de63f5d367d7a0bfd4e6702637fca789cbb4 WatchSource:0}: Error finding container ad11e55e213227150957f4065088de63f5d367d7a0bfd4e6702637fca789cbb4: Status 404 returned error can't find the container with id ad11e55e213227150957f4065088de63f5d367d7a0bfd4e6702637fca789cbb4
Jul 13 22:06:22.826794 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:22.826753    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx" event={"ID":"0023c91c-ea2f-4f3d-ae34-d31c4a8a842b","Type":"ContainerStarted","Data":"a1969e1232ab28d228817985d923cb2aa88e47d868f431d186bc89396ad01b6a"}
Jul 13 22:06:22.827182 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:22.826820    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx" event={"ID":"0023c91c-ea2f-4f3d-ae34-d31c4a8a842b","Type":"ContainerStarted","Data":"ad11e55e213227150957f4065088de63f5d367d7a0bfd4e6702637fca789cbb4"}
Jul 13 22:06:22.845923 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:22.845870    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx" podStartSLOduration=1.8458548179999998 podStartE2EDuration="1.845854818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:06:22.843883044 +0000 UTC m=+2224.000276884" watchObservedRunningTime="2026-07-13 22:06:22.845854818 +0000 UTC m=+2224.002248659"
Jul 13 22:06:23.832830 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:23.832798    2582 generic.go:356] "Generic (PLEG): container finished" podID="0023c91c-ea2f-4f3d-ae34-d31c4a8a842b" containerID="a1969e1232ab28d228817985d923cb2aa88e47d868f431d186bc89396ad01b6a" exitCode=0
Jul 13 22:06:23.833218 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:23.832844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx" event={"ID":"0023c91c-ea2f-4f3d-ae34-d31c4a8a842b","Type":"ContainerDied","Data":"a1969e1232ab28d228817985d923cb2aa88e47d868f431d186bc89396ad01b6a"}
Jul 13 22:06:24.969823 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:24.969797    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx"
Jul 13 22:06:25.029261 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:25.029228    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0023c91c-ea2f-4f3d-ae34-d31c4a8a842b-maas-api-service-ca\") pod \"0023c91c-ea2f-4f3d-ae34-d31c4a8a842b\" (UID: \"0023c91c-ea2f-4f3d-ae34-d31c4a8a842b\") "
Jul 13 22:06:25.029543 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:25.029521    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0023c91c-ea2f-4f3d-ae34-d31c4a8a842b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0023c91c-ea2f-4f3d-ae34-d31c4a8a842b" (UID: "0023c91c-ea2f-4f3d-ae34-d31c4a8a842b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:06:25.130091 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:25.130024    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0023c91c-ea2f-4f3d-ae34-d31c4a8a842b-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:06:25.843298 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:25.843266    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx"
Jul 13 22:06:25.843465 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:25.843303    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06fcd095-vzggx" event={"ID":"0023c91c-ea2f-4f3d-ae34-d31c4a8a842b","Type":"ContainerDied","Data":"ad11e55e213227150957f4065088de63f5d367d7a0bfd4e6702637fca789cbb4"}
Jul 13 22:06:25.843465 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:25.843367    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad11e55e213227150957f4065088de63f5d367d7a0bfd4e6702637fca789cbb4"
Jul 13 22:06:32.284103 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.284073    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"]
Jul 13 22:06:32.284534 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.284272    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr" podUID="e678cff5-7fd4-4303-9753-d3a36266d7cf" containerName="maas-api" containerID="cri-o://de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036" gracePeriod=30
Jul 13 22:06:32.313907 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.313876    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:06:32.533044 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.533019    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:06:32.597269 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.597192    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njqj7\" (UniqueName: \"kubernetes.io/projected/e678cff5-7fd4-4303-9753-d3a36266d7cf-kube-api-access-njqj7\") pod \"e678cff5-7fd4-4303-9753-d3a36266d7cf\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") "
Jul 13 22:06:32.597387 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.597294    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678cff5-7fd4-4303-9753-d3a36266d7cf-maas-api-tls\") pod \"e678cff5-7fd4-4303-9753-d3a36266d7cf\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") "
Jul 13 22:06:32.597387 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.597335    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e678cff5-7fd4-4303-9753-d3a36266d7cf-openshift-service-ca\") pod \"e678cff5-7fd4-4303-9753-d3a36266d7cf\" (UID: \"e678cff5-7fd4-4303-9753-d3a36266d7cf\") "
Jul 13 22:06:32.597688 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.597661    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e678cff5-7fd4-4303-9753-d3a36266d7cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e678cff5-7fd4-4303-9753-d3a36266d7cf" (UID: "e678cff5-7fd4-4303-9753-d3a36266d7cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:06:32.599229 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.599208    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e678cff5-7fd4-4303-9753-d3a36266d7cf-kube-api-access-njqj7" (OuterVolumeSpecName: "kube-api-access-njqj7") pod "e678cff5-7fd4-4303-9753-d3a36266d7cf" (UID: "e678cff5-7fd4-4303-9753-d3a36266d7cf"). InnerVolumeSpecName "kube-api-access-njqj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:06:32.599339 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.599321    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e678cff5-7fd4-4303-9753-d3a36266d7cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e678cff5-7fd4-4303-9753-d3a36266d7cf" (UID: "e678cff5-7fd4-4303-9753-d3a36266d7cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:06:32.698979 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.698951    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678cff5-7fd4-4303-9753-d3a36266d7cf-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:06:32.698979 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.698978    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e678cff5-7fd4-4303-9753-d3a36266d7cf-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:06:32.699146 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.698992    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njqj7\" (UniqueName: \"kubernetes.io/projected/e678cff5-7fd4-4303-9753-d3a36266d7cf-kube-api-access-njqj7\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:06:32.883717 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.883640    2582 generic.go:356] "Generic (PLEG): container finished" podID="e678cff5-7fd4-4303-9753-d3a36266d7cf" containerID="de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036" exitCode=0
Jul 13 22:06:32.883717 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.883684    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"
Jul 13 22:06:32.883983 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.883724    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr" event={"ID":"e678cff5-7fd4-4303-9753-d3a36266d7cf","Type":"ContainerDied","Data":"de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036"}
Jul 13 22:06:32.883983 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.883793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr" event={"ID":"e678cff5-7fd4-4303-9753-d3a36266d7cf","Type":"ContainerDied","Data":"e9cecbc0a4a3a1faab1f8afa8f2eef7de04c4727fffc804a10084ed2dd5994b2"}
Jul 13 22:06:32.883983 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.883826    2582 scope.go:117] "RemoveContainer" containerID="de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036"
Jul 13 22:06:32.893184 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.893169    2582 scope.go:117] "RemoveContainer" containerID="de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036"
Jul 13 22:06:32.893443 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:06:32.893420    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036\": container with ID starting with de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036 not found: ID does not exist" containerID="de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036"
Jul 13 22:06:32.893527 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.893447    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036"} err="failed to get container status \"de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036\": rpc error: code = NotFound desc = could not find container \"de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036\": container with ID starting with de7e8dc9731111c5ad2a8772948123ee87a068b406ca7701cb09e36c633cd036 not found: ID does not exist"
Jul 13 22:06:32.906828 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.906801    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"]
Jul 13 22:06:32.913576 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:32.913556    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fcd095-74bcdfb5bd-9hjqr"]
Jul 13 22:06:33.363790 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:33.363757    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e678cff5-7fd4-4303-9753-d3a36266d7cf" path="/var/lib/kubelet/pods/e678cff5-7fd4-4303-9753-d3a36266d7cf/volumes"
Jul 13 22:06:36.351692 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:36.351655    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:06:36.779591 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:36.779555    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:06:41.865192 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:06:41.865074    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:18.741579 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:18.741543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:19.281547 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:19.281511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:19.488551 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:19.488524    2582 scope.go:117] "RemoveContainer" containerID="30b954d08781012567f180a91068bae7289c16acf649c9584e9608906a6d2c3a"
Jul 13 22:07:24.860634 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:24.860592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:39.330571 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:39.330529    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:39.434450 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:39.434414    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:49.100559 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:49.100518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:49.309063 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:49.309027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:50.414503 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:50.414460    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:50.503515 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:50.503481    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:50.603539 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:50.603506    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:51.313326 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:51.313284    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:51.413588 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:51.413553    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:51.506890 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:51.506858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:52.921538 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:52.921504    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:07:53.191420 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:53.191333    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:07:58.762999 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:07:58.762958    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:00.293712 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:00.293675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:05.215805 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:05.215765    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:05.321079 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:05.321038    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:05.416105 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:05.416069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:07.156792 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:07.156754    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:08:15.969618 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:15.969535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:16.314316 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:16.314273    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:19.510094 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:19.510062    2582 scope.go:117] "RemoveContainer" containerID="17a560744aa5f7afdf6a2e425d7b4ad7091c2c404cae6e80cf4dbdfae01ece4d"
Jul 13 22:08:37.287265 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:37.287226    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:43.122561 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:43.122522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:43.188330 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:43.188296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:43.487884 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:43.487846    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:43.801798 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:43.801717    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:43.893357 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:43.893326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:43.993746 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:43.993716    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:44.149267 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:44.149166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:47.162377 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.162341    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:47.237441 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.237408    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"]
Jul 13 22:08:47.237785 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.237769    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e678cff5-7fd4-4303-9753-d3a36266d7cf" containerName="maas-api"
Jul 13 22:08:47.237785 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.237785    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e678cff5-7fd4-4303-9753-d3a36266d7cf" containerName="maas-api"
Jul 13 22:08:47.237938 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.237802    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0023c91c-ea2f-4f3d-ae34-d31c4a8a842b" containerName="revoke-keys"
Jul 13 22:08:47.237938 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.237811    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0023c91c-ea2f-4f3d-ae34-d31c4a8a842b" containerName="revoke-keys"
Jul 13 22:08:47.237938 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.237864    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e678cff5-7fd4-4303-9753-d3a36266d7cf" containerName="maas-api"
Jul 13 22:08:47.237938 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.237874    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0023c91c-ea2f-4f3d-ae34-d31c4a8a842b" containerName="revoke-keys"
Jul 13 22:08:47.241038 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.241008    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.244436 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.244400    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kfxsg\""
Jul 13 22:08:47.244789 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.244773    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 22:08:47.245422 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.245403    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-160510-serving-cert\""
Jul 13 22:08:47.245478 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.245457    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 22:08:47.251939 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.251915    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"]
Jul 13 22:08:47.253982 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.253874    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:47.331639 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.331614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:47.409039 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.409014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04da9c6a-fd41-411f-8b9d-d5c950906b46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-160510-785d5db47b-hqpfj\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.409223 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.409052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7l6\" (UniqueName: \"kubernetes.io/projected/04da9c6a-fd41-411f-8b9d-d5c950906b46-kube-api-access-7c7l6\") pod \"maas-api-e2e-inf-a-160510-785d5db47b-hqpfj\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.409223 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.409081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04da9c6a-fd41-411f-8b9d-d5c950906b46-maas-api-tls\") pod \"maas-api-e2e-inf-a-160510-785d5db47b-hqpfj\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.510288 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.510259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04da9c6a-fd41-411f-8b9d-d5c950906b46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-160510-785d5db47b-hqpfj\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.510475 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.510297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7l6\" (UniqueName: \"kubernetes.io/projected/04da9c6a-fd41-411f-8b9d-d5c950906b46-kube-api-access-7c7l6\") pod \"maas-api-e2e-inf-a-160510-785d5db47b-hqpfj\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.510475 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.510326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04da9c6a-fd41-411f-8b9d-d5c950906b46-maas-api-tls\") pod \"maas-api-e2e-inf-a-160510-785d5db47b-hqpfj\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.511245 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.511220    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04da9c6a-fd41-411f-8b9d-d5c950906b46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-160510-785d5db47b-hqpfj\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.513033 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.512989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04da9c6a-fd41-411f-8b9d-d5c950906b46-maas-api-tls\") pod \"maas-api-e2e-inf-a-160510-785d5db47b-hqpfj\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.513819 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.513792    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:47.520493 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.520469    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7l6\" (UniqueName: \"kubernetes.io/projected/04da9c6a-fd41-411f-8b9d-d5c950906b46-kube-api-access-7c7l6\") pod \"maas-api-e2e-inf-a-160510-785d5db47b-hqpfj\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.554254 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.554227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:47.698271 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.698247    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"]
Jul 13 22:08:47.699993 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:08:47.699949    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04da9c6a_fd41_411f_8b9d_d5c950906b46.slice/crio-28afa288588aea9e5db9f3b2e256b8ce0bc5977f5dfd823fe1dd73a9bece2965 WatchSource:0}: Error finding container 28afa288588aea9e5db9f3b2e256b8ce0bc5977f5dfd823fe1dd73a9bece2965: Status 404 returned error can't find the container with id 28afa288588aea9e5db9f3b2e256b8ce0bc5977f5dfd823fe1dd73a9bece2965
Jul 13 22:08:47.879338 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.879263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:47.976651 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:47.976623    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:48.539064 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:48.538986    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj" event={"ID":"04da9c6a-fd41-411f-8b9d-d5c950906b46","Type":"ContainerStarted","Data":"ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706"}
Jul 13 22:08:48.539064 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:48.539049    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj" event={"ID":"04da9c6a-fd41-411f-8b9d-d5c950906b46","Type":"ContainerStarted","Data":"28afa288588aea9e5db9f3b2e256b8ce0bc5977f5dfd823fe1dd73a9bece2965"}
Jul 13 22:08:48.539456 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:48.539167    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:48.557005 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:48.556961    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj" podStartSLOduration=0.998656507 podStartE2EDuration="1.556945565s" totalImagesPullingTime="558.289058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:08:48.556229525 +0000 UTC m=+2369.712623367" watchObservedRunningTime="2026-07-13 22:08:48.556945565 +0000 UTC m=+2369.713339406"
Jul 13 22:08:51.621327 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:51.621293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:51.943854 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:51.943762    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:52.259917 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:52.259879    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:52.654199 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:52.654090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:54.351715 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:54.351678    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:08:54.549259 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:54.549231    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:08:55.121886 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.121850    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:55.191940 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.191893    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"]
Jul 13 22:08:55.196022 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.195998    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:55.197879 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.197839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:55.198404 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.198383    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3394b0-serving-cert\""
Jul 13 22:08:55.208476 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.208455    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"]
Jul 13 22:08:55.272982 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.272946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94bfb49-01e9-4813-9616-7a681ed31711-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:55.273174 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.273046    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:55.273174 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.273106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gxkq\" (UniqueName: \"kubernetes.io/projected/c94bfb49-01e9-4813-9616-7a681ed31711-kube-api-access-8gxkq\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:55.273336 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.273317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:55.373775 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.373702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:55.373775 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.373736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94bfb49-01e9-4813-9616-7a681ed31711-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:55.374174 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.373777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gxkq\" (UniqueName: \"kubernetes.io/projected/c94bfb49-01e9-4813-9616-7a681ed31711-kube-api-access-8gxkq\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:55.374174 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:08:55.373853    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-serving-cert: secret "maas-api-e2e-inf-b-3394b0-serving-cert" not found
Jul 13 22:08:55.374174 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:08:55.373928    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls podName:c94bfb49-01e9-4813-9616-7a681ed31711 nodeName:}" failed. No retries permitted until 2026-07-13 22:08:55.873907318 +0000 UTC m=+2377.030301138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls") pod "maas-api-e2e-inf-b-3394b0-86599c889f-rng9d" (UID: "c94bfb49-01e9-4813-9616-7a681ed31711") : secret "maas-api-e2e-inf-b-3394b0-serving-cert" not found
Jul 13 22:08:55.374450 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.374432    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94bfb49-01e9-4813-9616-7a681ed31711-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:55.382789 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.382762    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gxkq\" (UniqueName: \"kubernetes.io/projected/c94bfb49-01e9-4813-9616-7a681ed31711-kube-api-access-8gxkq\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:55.706888 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.706772    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:55.799690 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.799660    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:55.878066 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.878031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:55.878256 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:08:55.878220    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-serving-cert: secret "maas-api-e2e-inf-b-3394b0-serving-cert" not found
Jul 13 22:08:55.878330 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:08:55.878298    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls podName:c94bfb49-01e9-4813-9616-7a681ed31711 nodeName:}" failed. No retries permitted until 2026-07-13 22:08:56.878276475 +0000 UTC m=+2378.034670298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls") pod "maas-api-e2e-inf-b-3394b0-86599c889f-rng9d" (UID: "c94bfb49-01e9-4813-9616-7a681ed31711") : secret "maas-api-e2e-inf-b-3394b0-serving-cert" not found
Jul 13 22:08:55.984921 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:55.984888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:08:56.885954 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:56.885913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:56.888323 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:56.888303    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls\") pod \"maas-api-e2e-inf-b-3394b0-86599c889f-rng9d\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:57.008889 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:57.008847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:57.342446 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:57.342421    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"]
Jul 13 22:08:57.344109 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:08:57.344084    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94bfb49_01e9_4813_9616_7a681ed31711.slice/crio-dcc8fffca46e3ea5b691cbd3cdb1a8a50b3723e86337d5d32105f3c9879c139c WatchSource:0}: Error finding container dcc8fffca46e3ea5b691cbd3cdb1a8a50b3723e86337d5d32105f3c9879c139c: Status 404 returned error can't find the container with id dcc8fffca46e3ea5b691cbd3cdb1a8a50b3723e86337d5d32105f3c9879c139c
Jul 13 22:08:57.585422 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:57.585391    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d" event={"ID":"c94bfb49-01e9-4813-9616-7a681ed31711","Type":"ContainerStarted","Data":"dcc8fffca46e3ea5b691cbd3cdb1a8a50b3723e86337d5d32105f3c9879c139c"}
Jul 13 22:08:58.592012 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:58.591975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d" event={"ID":"c94bfb49-01e9-4813-9616-7a681ed31711","Type":"ContainerStarted","Data":"ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0"}
Jul 13 22:08:58.592449 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:58.592095    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:08:58.610183 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:58.610120    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d" podStartSLOduration=3.288433032 podStartE2EDuration="3.610105251s" totalImagesPullingTime="321.672219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:08:58.608674691 +0000 UTC m=+2379.765068532" watchObservedRunningTime="2026-07-13 22:08:58.610105251 +0000 UTC m=+2379.766499101"
Jul 13 22:08:59.628116 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:08:59.628084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:04.602602 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:04.602569    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:09:05.457729 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:05.457692    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:09:08.714516 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:08.714480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:08.814340 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:08.814309    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:15.558688 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:15.558646    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:16.539410 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:16.539374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:16.634992 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:16.634964    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:18.753106 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:18.753071    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:09:19.388555 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:19.388528    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 22:09:19.393616 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:19.393583    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 22:09:19.396764 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:19.396745    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 22:09:19.401442 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:19.401425    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 22:09:29.251736 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:29.251699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:32.547011 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:32.546974    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:09:34.736715 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.736682    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"]
Jul 13 22:09:34.740337 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.740318    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:34.742837 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.742814    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3394b0\"/\"kube-root-ca.crt\""
Jul 13 22:09:34.743959 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.743936    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3394b0\"/\"default-dockercfg-k7h8b\""
Jul 13 22:09:34.744064 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.743943    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3394b0\"/\"openshift-service-ca.crt\""
Jul 13 22:09:34.744064 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.743995    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3394b0\"/\"test-model-3394b0-kserve-self-signed-certs\""
Jul 13 22:09:34.750518 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.750494    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"]
Jul 13 22:09:34.765714 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.765689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:34.929749 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.929720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tmp-dir\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:34.929894 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.929763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-dshm\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:34.929894 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.929787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkm4d\" (UniqueName: \"kubernetes.io/projected/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-kube-api-access-zkm4d\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:34.929894 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.929860    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-model-cache\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:34.929894 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.929878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tls-certs\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:34.930049 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:34.929896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-home\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.031239 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.031130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-model-cache\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.031239 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.031189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tls-certs\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.031239 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.031220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-home\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.031506 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.031253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tmp-dir\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.031506 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.031307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-dshm\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.031506 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.031344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkm4d\" (UniqueName: \"kubernetes.io/projected/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-kube-api-access-zkm4d\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.031665 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.031583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tmp-dir\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.031665 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.031638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-model-cache\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.031779 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.031680    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-home\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.033592 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.033559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-dshm\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.033704 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.033687    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tls-certs\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.040165 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.040127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkm4d\" (UniqueName: \"kubernetes.io/projected/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-kube-api-access-zkm4d\") pod \"test-model-3394b0-kserve-59dd89d67f-2c5ds\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") " pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.063537 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.063503    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:35.192492 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.192468    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"]
Jul 13 22:09:35.194633 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:09:35.194601    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3ee45f8_3d03_4a6a_93d8_fea824a9f660.slice/crio-1f8bf8f555b219194fdcc7dd0d41d97c34d78c3615f9614203a528572a8e9663 WatchSource:0}: Error finding container 1f8bf8f555b219194fdcc7dd0d41d97c34d78c3615f9614203a528572a8e9663: Status 404 returned error can't find the container with id 1f8bf8f555b219194fdcc7dd0d41d97c34d78c3615f9614203a528572a8e9663
Jul 13 22:09:35.779821 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.779783    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds" event={"ID":"e3ee45f8-3d03-4a6a-93d8-fea824a9f660","Type":"ContainerStarted","Data":"38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc"}
Jul 13 22:09:35.779821 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.779826    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds" event={"ID":"e3ee45f8-3d03-4a6a-93d8-fea824a9f660","Type":"ContainerStarted","Data":"1f8bf8f555b219194fdcc7dd0d41d97c34d78c3615f9614203a528572a8e9663"}
Jul 13 22:09:35.801912 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:35.801865    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds" podStartSLOduration=1.801850255 podStartE2EDuration="1.801850255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:09:35.799428959 +0000 UTC m=+2416.955822801" watchObservedRunningTime="2026-07-13 22:09:35.801850255 +0000 UTC m=+2416.958244096"
Jul 13 22:09:43.528076 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:43.527995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:45.064445 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:45.064408    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:45.064911 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:45.064465    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:45.078308 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:45.078285    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:45.838209 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:45.838176    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:09:48.762855 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:48.762820    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:09:52.999874 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:52.999833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:53.897467 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:53.897433    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"]
Jul 13 22:09:53.920112 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:53.920082    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:54.054585 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.054552    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls"]
Jul 13 22:09:54.058392 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.058374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls"
Jul 13 22:09:54.060911 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.060890    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:09:54.064426 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.064402    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls"]
Jul 13 22:09:54.179842 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.179765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beeb0b58-2f85-4aac-91d1-08aca9491e58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-160510-8flls\" (UID: \"beeb0b58-2f85-4aac-91d1-08aca9491e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls"
Jul 13 22:09:54.281265 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.281236    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beeb0b58-2f85-4aac-91d1-08aca9491e58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-160510-8flls\" (UID: \"beeb0b58-2f85-4aac-91d1-08aca9491e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls"
Jul 13 22:09:54.281825 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.281806    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beeb0b58-2f85-4aac-91d1-08aca9491e58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-160510-8flls\" (UID: \"beeb0b58-2f85-4aac-91d1-08aca9491e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls"
Jul 13 22:09:54.370597 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.370574    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls"
Jul 13 22:09:54.512587 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.512561    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls"]
Jul 13 22:09:54.514123 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:09:54.514099    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeeb0b58_2f85_4aac_91d1_08aca9491e58.slice/crio-275abfc68b672b01cb2fa7db7a6e8ea3d73fdceef5edabafb9fea19a701a4bd0 WatchSource:0}: Error finding container 275abfc68b672b01cb2fa7db7a6e8ea3d73fdceef5edabafb9fea19a701a4bd0: Status 404 returned error can't find the container with id 275abfc68b672b01cb2fa7db7a6e8ea3d73fdceef5edabafb9fea19a701a4bd0
Jul 13 22:09:54.638646 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.638615    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:54.733787 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.733766    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:09:54.872975 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.872944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls" event={"ID":"beeb0b58-2f85-4aac-91d1-08aca9491e58","Type":"ContainerStarted","Data":"c60a6edb4432f388644355f4595104237449d4abd9930b1d86430167a3b621eb"}
Jul 13 22:09:54.873127 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.872998    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls" event={"ID":"beeb0b58-2f85-4aac-91d1-08aca9491e58","Type":"ContainerStarted","Data":"275abfc68b672b01cb2fa7db7a6e8ea3d73fdceef5edabafb9fea19a701a4bd0"}
Jul 13 22:09:54.889771 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:54.889726    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls" podStartSLOduration=0.889707345 podStartE2EDuration="889.707345ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:09:54.888296982 +0000 UTC m=+2436.044690823" watchObservedRunningTime="2026-07-13 22:09:54.889707345 +0000 UTC m=+2436.046101186"
Jul 13 22:09:55.879895 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:55.879860    2582 generic.go:356] "Generic (PLEG): container finished" podID="beeb0b58-2f85-4aac-91d1-08aca9491e58" containerID="c60a6edb4432f388644355f4595104237449d4abd9930b1d86430167a3b621eb" exitCode=0
Jul 13 22:09:55.880283 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:55.879902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls" event={"ID":"beeb0b58-2f85-4aac-91d1-08aca9491e58","Type":"ContainerDied","Data":"c60a6edb4432f388644355f4595104237449d4abd9930b1d86430167a3b621eb"}
Jul 13 22:09:57.019062 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:57.019036    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls"
Jul 13 22:09:57.104169 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:57.104125    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beeb0b58-2f85-4aac-91d1-08aca9491e58-maas-api-service-ca\") pod \"beeb0b58-2f85-4aac-91d1-08aca9491e58\" (UID: \"beeb0b58-2f85-4aac-91d1-08aca9491e58\") "
Jul 13 22:09:57.104490 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:57.104466    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beeb0b58-2f85-4aac-91d1-08aca9491e58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "beeb0b58-2f85-4aac-91d1-08aca9491e58" (UID: "beeb0b58-2f85-4aac-91d1-08aca9491e58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:09:57.205682 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:57.205604    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beeb0b58-2f85-4aac-91d1-08aca9491e58-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:09:57.891368 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:57.891338    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls"
Jul 13 22:09:57.891542 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:57.891333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-160510-8flls" event={"ID":"beeb0b58-2f85-4aac-91d1-08aca9491e58","Type":"ContainerDied","Data":"275abfc68b672b01cb2fa7db7a6e8ea3d73fdceef5edabafb9fea19a701a4bd0"}
Jul 13 22:09:57.891542 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:09:57.891462    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="275abfc68b672b01cb2fa7db7a6e8ea3d73fdceef5edabafb9fea19a701a4bd0"
Jul 13 22:10:04.454768 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.454725    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"]
Jul 13 22:10:04.457443 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.455484    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj" podUID="04da9c6a-fd41-411f-8b9d-d5c950906b46" containerName="maas-api" containerID="cri-o://ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706" gracePeriod=30
Jul 13 22:10:04.479399 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.479370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:04.544546 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.544503    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj" podUID="04da9c6a-fd41-411f-8b9d-d5c950906b46" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.47:8443/health\": dial tcp 10.133.0.47:8443: connect: connection refused"
Jul 13 22:10:04.708356 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.708298    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:10:04.770318 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.770282    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c7l6\" (UniqueName: \"kubernetes.io/projected/04da9c6a-fd41-411f-8b9d-d5c950906b46-kube-api-access-7c7l6\") pod \"04da9c6a-fd41-411f-8b9d-d5c950906b46\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") "
Jul 13 22:10:04.770489 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.770338    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04da9c6a-fd41-411f-8b9d-d5c950906b46-openshift-service-ca\") pod \"04da9c6a-fd41-411f-8b9d-d5c950906b46\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") "
Jul 13 22:10:04.770489 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.770394    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04da9c6a-fd41-411f-8b9d-d5c950906b46-maas-api-tls\") pod \"04da9c6a-fd41-411f-8b9d-d5c950906b46\" (UID: \"04da9c6a-fd41-411f-8b9d-d5c950906b46\") "
Jul 13 22:10:04.770707 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.770682    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04da9c6a-fd41-411f-8b9d-d5c950906b46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04da9c6a-fd41-411f-8b9d-d5c950906b46" (UID: "04da9c6a-fd41-411f-8b9d-d5c950906b46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:10:04.772457 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.772428    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04da9c6a-fd41-411f-8b9d-d5c950906b46-kube-api-access-7c7l6" (OuterVolumeSpecName: "kube-api-access-7c7l6") pod "04da9c6a-fd41-411f-8b9d-d5c950906b46" (UID: "04da9c6a-fd41-411f-8b9d-d5c950906b46"). InnerVolumeSpecName "kube-api-access-7c7l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:10:04.772550 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.772532    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04da9c6a-fd41-411f-8b9d-d5c950906b46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04da9c6a-fd41-411f-8b9d-d5c950906b46" (UID: "04da9c6a-fd41-411f-8b9d-d5c950906b46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:10:04.871487 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.871456    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c7l6\" (UniqueName: \"kubernetes.io/projected/04da9c6a-fd41-411f-8b9d-d5c950906b46-kube-api-access-7c7l6\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:04.871487 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.871489    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04da9c6a-fd41-411f-8b9d-d5c950906b46-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:04.871686 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.871503    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04da9c6a-fd41-411f-8b9d-d5c950906b46-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:04.924541 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.924510    2582 generic.go:356] "Generic (PLEG): container finished" podID="04da9c6a-fd41-411f-8b9d-d5c950906b46" containerID="ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706" exitCode=0
Jul 13 22:10:04.924718 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.924557    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"
Jul 13 22:10:04.924718 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.924600    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj" event={"ID":"04da9c6a-fd41-411f-8b9d-d5c950906b46","Type":"ContainerDied","Data":"ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706"}
Jul 13 22:10:04.924718 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.924651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj" event={"ID":"04da9c6a-fd41-411f-8b9d-d5c950906b46","Type":"ContainerDied","Data":"28afa288588aea9e5db9f3b2e256b8ce0bc5977f5dfd823fe1dd73a9bece2965"}
Jul 13 22:10:04.924718 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.924674    2582 scope.go:117] "RemoveContainer" containerID="ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706"
Jul 13 22:10:04.935241 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.935188    2582 scope.go:117] "RemoveContainer" containerID="ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706"
Jul 13 22:10:04.935649 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:10:04.935592    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706\": container with ID starting with ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706 not found: ID does not exist" containerID="ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706"
Jul 13 22:10:04.935649 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.935624    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706"} err="failed to get container status \"ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706\": rpc error: code = NotFound desc = could not find container \"ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706\": container with ID starting with ef510888055660851a163c0a86f2fffcb03b70fa09de790414a67a7b86da2706 not found: ID does not exist"
Jul 13 22:10:04.947744 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.947713    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"]
Jul 13 22:10:04.949567 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:04.949547    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-160510-785d5db47b-hqpfj"]
Jul 13 22:10:05.363881 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:05.363846    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04da9c6a-fd41-411f-8b9d-d5c950906b46" path="/var/lib/kubelet/pods/04da9c6a-fd41-411f-8b9d-d5c950906b46/volumes"
Jul 13 22:10:06.851482 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:06.851436    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:10:08.917600 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:08.917561    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds" podUID="e3ee45f8-3d03-4a6a-93d8-fea824a9f660" containerName="main" containerID="cri-o://38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc" gracePeriod=45
Jul 13 22:10:09.142495 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.142456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:09.166507 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.166481    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:10:09.207522 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.207442    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:09.309432 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.309399    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkm4d\" (UniqueName: \"kubernetes.io/projected/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-kube-api-access-zkm4d\") pod \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") "
Jul 13 22:10:09.309616 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.309444    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tls-certs\") pod \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") "
Jul 13 22:10:09.309616 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.309480    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-model-cache\") pod \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") "
Jul 13 22:10:09.309616 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.309502    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-dshm\") pod \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") "
Jul 13 22:10:09.309616 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.309589    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-home\") pod \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") "
Jul 13 22:10:09.309840 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.309638    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tmp-dir\") pod \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\" (UID: \"e3ee45f8-3d03-4a6a-93d8-fea824a9f660\") "
Jul 13 22:10:09.309840 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.309796    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3ee45f8-3d03-4a6a-93d8-fea824a9f660" (UID: "e3ee45f8-3d03-4a6a-93d8-fea824a9f660"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:10:09.309945 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.309918    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-home" (OuterVolumeSpecName: "home") pod "e3ee45f8-3d03-4a6a-93d8-fea824a9f660" (UID: "e3ee45f8-3d03-4a6a-93d8-fea824a9f660"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:10:09.309945 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.309932    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-model-cache\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:09.310024 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.309943    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3ee45f8-3d03-4a6a-93d8-fea824a9f660" (UID: "e3ee45f8-3d03-4a6a-93d8-fea824a9f660"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:10:09.311730 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.311705    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-dshm" (OuterVolumeSpecName: "dshm") pod "e3ee45f8-3d03-4a6a-93d8-fea824a9f660" (UID: "e3ee45f8-3d03-4a6a-93d8-fea824a9f660"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:10:09.311830 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.311768    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3ee45f8-3d03-4a6a-93d8-fea824a9f660" (UID: "e3ee45f8-3d03-4a6a-93d8-fea824a9f660"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:10:09.311830 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.311795    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-kube-api-access-zkm4d" (OuterVolumeSpecName: "kube-api-access-zkm4d") pod "e3ee45f8-3d03-4a6a-93d8-fea824a9f660" (UID: "e3ee45f8-3d03-4a6a-93d8-fea824a9f660"). InnerVolumeSpecName "kube-api-access-zkm4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:10:09.410994 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.410962    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tmp-dir\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:09.410994 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.410990    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkm4d\" (UniqueName: \"kubernetes.io/projected/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-kube-api-access-zkm4d\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:09.411174 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.411000    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-tls-certs\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:09.411174 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.411009    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-dshm\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:09.411174 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.411017    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ee45f8-3d03-4a6a-93d8-fea824a9f660-home\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:09.950400 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.950365    2582 generic.go:356] "Generic (PLEG): container finished" podID="e3ee45f8-3d03-4a6a-93d8-fea824a9f660" containerID="38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc" exitCode=0
Jul 13 22:10:09.950824 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.950406    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"
Jul 13 22:10:09.950824 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.950420    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds" event={"ID":"e3ee45f8-3d03-4a6a-93d8-fea824a9f660","Type":"ContainerDied","Data":"38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc"}
Jul 13 22:10:09.950824 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.950459    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds" event={"ID":"e3ee45f8-3d03-4a6a-93d8-fea824a9f660","Type":"ContainerDied","Data":"1f8bf8f555b219194fdcc7dd0d41d97c34d78c3615f9614203a528572a8e9663"}
Jul 13 22:10:09.950824 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.950480    2582 scope.go:117] "RemoveContainer" containerID="38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc"
Jul 13 22:10:09.959059 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.959039    2582 scope.go:117] "RemoveContainer" containerID="38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc"
Jul 13 22:10:09.959321 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:10:09.959303    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc\": container with ID starting with 38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc not found: ID does not exist" containerID="38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc"
Jul 13 22:10:09.959401 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.959335    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc"} err="failed to get container status \"38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc\": rpc error: code = NotFound desc = could not find container \"38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc\": container with ID starting with 38f99a5f89980d83e9e07528bf87090419f669d77fa6136622f074040809bbdc not found: ID does not exist"
Jul 13 22:10:09.967987 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.967967    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"]
Jul 13 22:10:09.971808 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:09.971790    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3394b0/test-model-3394b0-kserve-59dd89d67f-2c5ds"]
Jul 13 22:10:11.363661 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:11.363630    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ee45f8-3d03-4a6a-93d8-fea824a9f660" path="/var/lib/kubelet/pods/e3ee45f8-3d03-4a6a-93d8-fea824a9f660/volumes"
Jul 13 22:10:15.350902 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:15.350862    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:15.451323 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:15.451290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:19.849856 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:19.849762    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:20.084308 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.084266    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:20.196367 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.196286    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj"]
Jul 13 22:10:20.196642 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.196629    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04da9c6a-fd41-411f-8b9d-d5c950906b46" containerName="maas-api"
Jul 13 22:10:20.196685 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.196644    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="04da9c6a-fd41-411f-8b9d-d5c950906b46" containerName="maas-api"
Jul 13 22:10:20.196685 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.196661    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ee45f8-3d03-4a6a-93d8-fea824a9f660" containerName="main"
Jul 13 22:10:20.196685 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.196666    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ee45f8-3d03-4a6a-93d8-fea824a9f660" containerName="main"
Jul 13 22:10:20.196685 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.196677    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beeb0b58-2f85-4aac-91d1-08aca9491e58" containerName="revoke-keys"
Jul 13 22:10:20.196685 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.196683    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="beeb0b58-2f85-4aac-91d1-08aca9491e58" containerName="revoke-keys"
Jul 13 22:10:20.196833 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.196734    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="04da9c6a-fd41-411f-8b9d-d5c950906b46" containerName="maas-api"
Jul 13 22:10:20.196833 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.196742    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ee45f8-3d03-4a6a-93d8-fea824a9f660" containerName="main"
Jul 13 22:10:20.196833 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.196749    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="beeb0b58-2f85-4aac-91d1-08aca9491e58" containerName="revoke-keys"
Jul 13 22:10:20.201366 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.201345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj"
Jul 13 22:10:20.204192 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.204168    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:10:20.207607 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.207581    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj"]
Jul 13 22:10:20.312322 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.312283    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cb5891-2e79-40f8-b956-dde8f7e3d4c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj\" (UID: \"b2cb5891-2e79-40f8-b956-dde8f7e3d4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj"
Jul 13 22:10:20.413522 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.413484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cb5891-2e79-40f8-b956-dde8f7e3d4c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj\" (UID: \"b2cb5891-2e79-40f8-b956-dde8f7e3d4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj"
Jul 13 22:10:20.414127 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.414106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cb5891-2e79-40f8-b956-dde8f7e3d4c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj\" (UID: \"b2cb5891-2e79-40f8-b956-dde8f7e3d4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj"
Jul 13 22:10:20.512910 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.512884    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj"
Jul 13 22:10:20.643611 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.643586    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj"]
Jul 13 22:10:20.645992 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:10:20.645962    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cb5891_2e79_40f8_b956_dde8f7e3d4c3.slice/crio-9ed6ec0359b0a237db0803ff04ea95c52c1d1eb60f4e6980b5e265a58fc6c3be WatchSource:0}: Error finding container 9ed6ec0359b0a237db0803ff04ea95c52c1d1eb60f4e6980b5e265a58fc6c3be: Status 404 returned error can't find the container with id 9ed6ec0359b0a237db0803ff04ea95c52c1d1eb60f4e6980b5e265a58fc6c3be
Jul 13 22:10:20.716902 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.716872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:20.819042 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:20.818955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:21.006088 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:21.006053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj" event={"ID":"b2cb5891-2e79-40f8-b956-dde8f7e3d4c3","Type":"ContainerStarted","Data":"319a1c3bae3a7c5c3dbb7b42a3ea8223abe257b5aea5379cd3434f5426c34aec"}
Jul 13 22:10:21.006502 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:21.006146    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj" event={"ID":"b2cb5891-2e79-40f8-b956-dde8f7e3d4c3","Type":"ContainerStarted","Data":"9ed6ec0359b0a237db0803ff04ea95c52c1d1eb60f4e6980b5e265a58fc6c3be"}
Jul 13 22:10:21.024543 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:21.024498    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj" podStartSLOduration=1.024482594 podStartE2EDuration="1.024482594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:10:21.022921818 +0000 UTC m=+2462.179315661" watchObservedRunningTime="2026-07-13 22:10:21.024482594 +0000 UTC m=+2462.180876435"
Jul 13 22:10:22.011573 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:22.011533    2582 generic.go:356] "Generic (PLEG): container finished" podID="b2cb5891-2e79-40f8-b956-dde8f7e3d4c3" containerID="319a1c3bae3a7c5c3dbb7b42a3ea8223abe257b5aea5379cd3434f5426c34aec" exitCode=0
Jul 13 22:10:22.011969 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:22.011625    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj" event={"ID":"b2cb5891-2e79-40f8-b956-dde8f7e3d4c3","Type":"ContainerDied","Data":"319a1c3bae3a7c5c3dbb7b42a3ea8223abe257b5aea5379cd3434f5426c34aec"}
Jul 13 22:10:23.145592 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:23.145568    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj"
Jul 13 22:10:23.236693 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:23.236663    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cb5891-2e79-40f8-b956-dde8f7e3d4c3-maas-api-service-ca\") pod \"b2cb5891-2e79-40f8-b956-dde8f7e3d4c3\" (UID: \"b2cb5891-2e79-40f8-b956-dde8f7e3d4c3\") "
Jul 13 22:10:23.236979 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:23.236957    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2cb5891-2e79-40f8-b956-dde8f7e3d4c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2cb5891-2e79-40f8-b956-dde8f7e3d4c3" (UID: "b2cb5891-2e79-40f8-b956-dde8f7e3d4c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:10:23.337965 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:23.337906    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cb5891-2e79-40f8-b956-dde8f7e3d4c3-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:24.021684 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:24.021649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj" event={"ID":"b2cb5891-2e79-40f8-b956-dde8f7e3d4c3","Type":"ContainerDied","Data":"9ed6ec0359b0a237db0803ff04ea95c52c1d1eb60f4e6980b5e265a58fc6c3be"}
Jul 13 22:10:24.021684 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:24.021683    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj"
Jul 13 22:10:24.021971 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:24.021709    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ed6ec0359b0a237db0803ff04ea95c52c1d1eb60f4e6980b5e265a58fc6c3be"
Jul 13 22:10:30.601723 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:30.601687    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"]
Jul 13 22:10:30.602140 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:30.601949    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d" podUID="c94bfb49-01e9-4813-9616-7a681ed31711" containerName="maas-api" containerID="cri-o://ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0" gracePeriod=30
Jul 13 22:10:30.626292 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:30.626261    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:30.852810 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:30.852756    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:10:31.002481 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.002451    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94bfb49-01e9-4813-9616-7a681ed31711-openshift-service-ca\") pod \"c94bfb49-01e9-4813-9616-7a681ed31711\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") "
Jul 13 22:10:31.002666 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.002497    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls\") pod \"c94bfb49-01e9-4813-9616-7a681ed31711\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") "
Jul 13 22:10:31.002666 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.002527    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gxkq\" (UniqueName: \"kubernetes.io/projected/c94bfb49-01e9-4813-9616-7a681ed31711-kube-api-access-8gxkq\") pod \"c94bfb49-01e9-4813-9616-7a681ed31711\" (UID: \"c94bfb49-01e9-4813-9616-7a681ed31711\") "
Jul 13 22:10:31.002874 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.002848    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94bfb49-01e9-4813-9616-7a681ed31711-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c94bfb49-01e9-4813-9616-7a681ed31711" (UID: "c94bfb49-01e9-4813-9616-7a681ed31711"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:10:31.004667 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.004645    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94bfb49-01e9-4813-9616-7a681ed31711-kube-api-access-8gxkq" (OuterVolumeSpecName: "kube-api-access-8gxkq") pod "c94bfb49-01e9-4813-9616-7a681ed31711" (UID: "c94bfb49-01e9-4813-9616-7a681ed31711"). InnerVolumeSpecName "kube-api-access-8gxkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:10:31.004733 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.004658    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c94bfb49-01e9-4813-9616-7a681ed31711" (UID: "c94bfb49-01e9-4813-9616-7a681ed31711"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:10:31.055292 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.055266    2582 generic.go:356] "Generic (PLEG): container finished" podID="c94bfb49-01e9-4813-9616-7a681ed31711" containerID="ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0" exitCode=0
Jul 13 22:10:31.055387 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.055307    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"
Jul 13 22:10:31.055387 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.055355    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d" event={"ID":"c94bfb49-01e9-4813-9616-7a681ed31711","Type":"ContainerDied","Data":"ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0"}
Jul 13 22:10:31.055463 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.055404    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d" event={"ID":"c94bfb49-01e9-4813-9616-7a681ed31711","Type":"ContainerDied","Data":"dcc8fffca46e3ea5b691cbd3cdb1a8a50b3723e86337d5d32105f3c9879c139c"}
Jul 13 22:10:31.055463 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.055425    2582 scope.go:117] "RemoveContainer" containerID="ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0"
Jul 13 22:10:31.063824 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.063807    2582 scope.go:117] "RemoveContainer" containerID="ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0"
Jul 13 22:10:31.064058 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:10:31.064041    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0\": container with ID starting with ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0 not found: ID does not exist" containerID="ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0"
Jul 13 22:10:31.064111 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.064063    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0"} err="failed to get container status \"ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0\": rpc error: code = NotFound desc = could not find container \"ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0\": container with ID starting with ea69b0aef322499a9b859d5afed059507472612798b75be4e34cfc383f0eddc0 not found: ID does not exist"
Jul 13 22:10:31.078812 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.078788    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"]
Jul 13 22:10:31.084966 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.084945    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3394b0-86599c889f-rng9d"]
Jul 13 22:10:31.104121 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.104077    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gxkq\" (UniqueName: \"kubernetes.io/projected/c94bfb49-01e9-4813-9616-7a681ed31711-kube-api-access-8gxkq\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:31.104121 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.104098    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94bfb49-01e9-4813-9616-7a681ed31711-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:31.104121 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.104110    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94bfb49-01e9-4813-9616-7a681ed31711-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:10:31.363879 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:31.363819    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c94bfb49-01e9-4813-9616-7a681ed31711" path="/var/lib/kubelet/pods/c94bfb49-01e9-4813-9616-7a681ed31711/volumes"
Jul 13 22:10:33.847465 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:33.847416    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:10:35.285091 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:35.285044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:41.183867 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:41.183826    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:41.294344 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:41.294308    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:45.978086 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:45.978045    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:46.214808 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:46.214762    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:46.569705 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:46.569663    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:46.878733 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:46.878644    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:47.637015 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:47.636980    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:49.818227 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.818187    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:49.950867 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.950836    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"]
Jul 13 22:10:49.951201 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.951186    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c94bfb49-01e9-4813-9616-7a681ed31711" containerName="maas-api"
Jul 13 22:10:49.951249 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.951202    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c94bfb49-01e9-4813-9616-7a681ed31711" containerName="maas-api"
Jul 13 22:10:49.951249 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.951215    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2cb5891-2e79-40f8-b956-dde8f7e3d4c3" containerName="revoke-keys"
Jul 13 22:10:49.951249 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.951221    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2cb5891-2e79-40f8-b956-dde8f7e3d4c3" containerName="revoke-keys"
Jul 13 22:10:49.951343 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.951272    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c94bfb49-01e9-4813-9616-7a681ed31711" containerName="maas-api"
Jul 13 22:10:49.951343 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.951283    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2cb5891-2e79-40f8-b956-dde8f7e3d4c3" containerName="revoke-keys"
Jul 13 22:10:49.955838 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.955816    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:49.958680 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.958654    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-785879-serving-cert\""
Jul 13 22:10:49.958822 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.958738    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 22:10:49.959663 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.959634    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kfxsg\""
Jul 13 22:10:49.959663 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.959636    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 22:10:49.965936 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:49.965909    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"]
Jul 13 22:10:50.007723 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.007697    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:50.055191 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.055129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79aab0ac-1575-4ac6-9185-19ad53b37acf-maas-api-tls\") pod \"maas-api-e2e-api-a-785879-8588dfb954-xn2s6\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:50.055191 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.055199    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79aab0ac-1575-4ac6-9185-19ad53b37acf-openshift-service-ca\") pod \"maas-api-e2e-api-a-785879-8588dfb954-xn2s6\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:50.055375 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.055232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-996zr\" (UniqueName: \"kubernetes.io/projected/79aab0ac-1575-4ac6-9185-19ad53b37acf-kube-api-access-996zr\") pod \"maas-api-e2e-api-a-785879-8588dfb954-xn2s6\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:50.104941 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.104870    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:50.156607 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.156579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79aab0ac-1575-4ac6-9185-19ad53b37acf-maas-api-tls\") pod \"maas-api-e2e-api-a-785879-8588dfb954-xn2s6\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:50.156732 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.156614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79aab0ac-1575-4ac6-9185-19ad53b37acf-openshift-service-ca\") pod \"maas-api-e2e-api-a-785879-8588dfb954-xn2s6\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:50.156732 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.156636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-996zr\" (UniqueName: \"kubernetes.io/projected/79aab0ac-1575-4ac6-9185-19ad53b37acf-kube-api-access-996zr\") pod \"maas-api-e2e-api-a-785879-8588dfb954-xn2s6\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:50.157369 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.157349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79aab0ac-1575-4ac6-9185-19ad53b37acf-openshift-service-ca\") pod \"maas-api-e2e-api-a-785879-8588dfb954-xn2s6\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:50.159016 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.158998    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79aab0ac-1575-4ac6-9185-19ad53b37acf-maas-api-tls\") pod \"maas-api-e2e-api-a-785879-8588dfb954-xn2s6\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:50.167342 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.167319    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-996zr\" (UniqueName: \"kubernetes.io/projected/79aab0ac-1575-4ac6-9185-19ad53b37acf-kube-api-access-996zr\") pod \"maas-api-e2e-api-a-785879-8588dfb954-xn2s6\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:50.202745 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.202713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:50.267165 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.267135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:50.399234 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.399193    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"]
Jul 13 22:10:50.402187 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:10:50.402143    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79aab0ac_1575_4ac6_9185_19ad53b37acf.slice/crio-66d64d6bdfcede252c65e4c9d5f9b89d70a288ca4b9d7cabe892515405ff58a7 WatchSource:0}: Error finding container 66d64d6bdfcede252c65e4c9d5f9b89d70a288ca4b9d7cabe892515405ff58a7: Status 404 returned error can't find the container with id 66d64d6bdfcede252c65e4c9d5f9b89d70a288ca4b9d7cabe892515405ff58a7
Jul 13 22:10:50.403594 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.403575    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 22:10:50.454650 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.454613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:50.560166 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:50.560113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:51.156119 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:51.156022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6" event={"ID":"79aab0ac-1575-4ac6-9185-19ad53b37acf","Type":"ContainerStarted","Data":"d11a9a150f7d5a32d3097a39a2bb5550932e9e56383ded607e12ab9fad51d9da"}
Jul 13 22:10:51.156501 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:51.156118    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:51.156501 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:51.156145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6" event={"ID":"79aab0ac-1575-4ac6-9185-19ad53b37acf","Type":"ContainerStarted","Data":"66d64d6bdfcede252c65e4c9d5f9b89d70a288ca4b9d7cabe892515405ff58a7"}
Jul 13 22:10:51.177747 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:51.177697    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6" podStartSLOduration=1.725406216 podStartE2EDuration="2.177682953s" totalImagesPullingTime="452.276737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:10:51.175774824 +0000 UTC m=+2492.332168668" watchObservedRunningTime="2026-07-13 22:10:51.177682953 +0000 UTC m=+2492.334076794"
Jul 13 22:10:54.313360 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:54.313320    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:54.446922 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:54.446884    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:10:54.753669 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:54.753627    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:55.105897 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:55.105818    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:55.681866 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:55.681810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:57.166668 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:57.166634    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:10:57.910629 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:57.910574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:57.990353 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:57.990325    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"]
Jul 13 22:10:57.993098 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:57.993065    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:57.993602 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:57.993581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:57.995958 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:57.995935    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9dff5a-serving-cert\""
Jul 13 22:10:58.004990 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.004969    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"]
Jul 13 22:10:58.077551 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.077517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:58.127784 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.127750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e49acc-65a4-4cbe-801b-fc050e2407c3-maas-api-tls\") pod \"maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:58.127898 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.127784    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6js5s\" (UniqueName: \"kubernetes.io/projected/70e49acc-65a4-4cbe-801b-fc050e2407c3-kube-api-access-6js5s\") pod \"maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:58.127942 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.127887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e49acc-65a4-4cbe-801b-fc050e2407c3-openshift-service-ca\") pod \"maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:58.216732 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.216639    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:58.228746 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.228719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e49acc-65a4-4cbe-801b-fc050e2407c3-openshift-service-ca\") pod \"maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:58.228844 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.228770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e49acc-65a4-4cbe-801b-fc050e2407c3-maas-api-tls\") pod \"maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:58.228844 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.228789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6js5s\" (UniqueName: \"kubernetes.io/projected/70e49acc-65a4-4cbe-801b-fc050e2407c3-kube-api-access-6js5s\") pod \"maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:58.229429 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.229407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e49acc-65a4-4cbe-801b-fc050e2407c3-openshift-service-ca\") pod \"maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:58.231265 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.231244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e49acc-65a4-4cbe-801b-fc050e2407c3-maas-api-tls\") pod \"maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:58.239782 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.239757    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6js5s\" (UniqueName: \"kubernetes.io/projected/70e49acc-65a4-4cbe-801b-fc050e2407c3-kube-api-access-6js5s\") pod \"maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:58.304601 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.304581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:58.440141 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.440117    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"]
Jul 13 22:10:58.442043 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:10:58.442019    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e49acc_65a4_4cbe_801b_fc050e2407c3.slice/crio-8aa291c183b563ca22a7f2126782cf2861f491dba3ddd1d69724bc3e68bf79be WatchSource:0}: Error finding container 8aa291c183b563ca22a7f2126782cf2861f491dba3ddd1d69724bc3e68bf79be: Status 404 returned error can't find the container with id 8aa291c183b563ca22a7f2126782cf2861f491dba3ddd1d69724bc3e68bf79be
Jul 13 22:10:58.474969 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.474909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:58.580715 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:58.580689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:10:59.196019 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:59.195928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k" event={"ID":"70e49acc-65a4-4cbe-801b-fc050e2407c3","Type":"ContainerStarted","Data":"c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a"}
Jul 13 22:10:59.196019 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:59.195982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k" event={"ID":"70e49acc-65a4-4cbe-801b-fc050e2407c3","Type":"ContainerStarted","Data":"8aa291c183b563ca22a7f2126782cf2861f491dba3ddd1d69724bc3e68bf79be"}
Jul 13 22:10:59.196019 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:59.196015    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:10:59.216098 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:10:59.216050    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k" podStartSLOduration=1.813066313 podStartE2EDuration="2.216035907s" totalImagesPullingTime="402.969594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:10:59.214687568 +0000 UTC m=+2500.371081410" watchObservedRunningTime="2026-07-13 22:10:59.216035907 +0000 UTC m=+2500.372429747"
Jul 13 22:11:02.322944 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:02.322909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:04.979405 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:04.979369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:05.083604 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:05.083566    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:05.206548 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:05.206512    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:11:06.551252 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:06.551219    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:11:10.387596 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.387559    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb"]
Jul 13 22:11:10.391237 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.391217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb"
Jul 13 22:11:10.393999 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.393978    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:11:10.400614 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.400590    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb"]
Jul 13 22:11:10.538727 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.538693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912ba5d2-ec45-4a89-a8d3-7159bf7f59e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-785879-l88xb\" (UID: \"912ba5d2-ec45-4a89-a8d3-7159bf7f59e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb"
Jul 13 22:11:10.639683 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.639582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912ba5d2-ec45-4a89-a8d3-7159bf7f59e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-785879-l88xb\" (UID: \"912ba5d2-ec45-4a89-a8d3-7159bf7f59e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb"
Jul 13 22:11:10.640241 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.640219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912ba5d2-ec45-4a89-a8d3-7159bf7f59e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-785879-l88xb\" (UID: \"912ba5d2-ec45-4a89-a8d3-7159bf7f59e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb"
Jul 13 22:11:10.703482 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.703459    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb"
Jul 13 22:11:10.834222 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.834175    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb"]
Jul 13 22:11:10.837690 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:11:10.837664    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912ba5d2_ec45_4a89_a8d3_7159bf7f59e8.slice/crio-00cb495f29042762b055410ca828011efe9288893a9979b8a1afe517e306e38f WatchSource:0}: Error finding container 00cb495f29042762b055410ca828011efe9288893a9979b8a1afe517e306e38f: Status 404 returned error can't find the container with id 00cb495f29042762b055410ca828011efe9288893a9979b8a1afe517e306e38f
Jul 13 22:11:10.877137 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.877101    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:10.979779 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:10.979754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:11.257827 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:11.257732    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb" event={"ID":"912ba5d2-ec45-4a89-a8d3-7159bf7f59e8","Type":"ContainerStarted","Data":"c228e86606ef9e761d9dedf94b9a88c79415c474931d76a34dd0d48e218b1465"}
Jul 13 22:11:11.257827 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:11.257781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb" event={"ID":"912ba5d2-ec45-4a89-a8d3-7159bf7f59e8","Type":"ContainerStarted","Data":"00cb495f29042762b055410ca828011efe9288893a9979b8a1afe517e306e38f"}
Jul 13 22:11:11.273281 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:11.273237    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb" podStartSLOduration=1.273221544 podStartE2EDuration="1.273221544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:11:11.272191083 +0000 UTC m=+2512.428584926" watchObservedRunningTime="2026-07-13 22:11:11.273221544 +0000 UTC m=+2512.429615363"
Jul 13 22:11:12.263422 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:12.263391    2582 generic.go:356] "Generic (PLEG): container finished" podID="912ba5d2-ec45-4a89-a8d3-7159bf7f59e8" containerID="c228e86606ef9e761d9dedf94b9a88c79415c474931d76a34dd0d48e218b1465" exitCode=0
Jul 13 22:11:12.263853 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:12.263491    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb" event={"ID":"912ba5d2-ec45-4a89-a8d3-7159bf7f59e8","Type":"ContainerDied","Data":"c228e86606ef9e761d9dedf94b9a88c79415c474931d76a34dd0d48e218b1465"}
Jul 13 22:11:13.411635 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:13.411613    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb"
Jul 13 22:11:13.565708 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:13.565634    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912ba5d2-ec45-4a89-a8d3-7159bf7f59e8-maas-api-service-ca\") pod \"912ba5d2-ec45-4a89-a8d3-7159bf7f59e8\" (UID: \"912ba5d2-ec45-4a89-a8d3-7159bf7f59e8\") "
Jul 13 22:11:13.565978 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:13.565956    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912ba5d2-ec45-4a89-a8d3-7159bf7f59e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "912ba5d2-ec45-4a89-a8d3-7159bf7f59e8" (UID: "912ba5d2-ec45-4a89-a8d3-7159bf7f59e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:11:13.667120 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:13.667088    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912ba5d2-ec45-4a89-a8d3-7159bf7f59e8-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:11:14.106008 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:14.105978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:14.275257 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:14.275228    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb"
Jul 13 22:11:14.275413 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:14.275226    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-785879-l88xb" event={"ID":"912ba5d2-ec45-4a89-a8d3-7159bf7f59e8","Type":"ContainerDied","Data":"00cb495f29042762b055410ca828011efe9288893a9979b8a1afe517e306e38f"}
Jul 13 22:11:14.275413 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:14.275352    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00cb495f29042762b055410ca828011efe9288893a9979b8a1afe517e306e38f"
Jul 13 22:11:19.288915 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:19.288845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:19.390000 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:19.389954    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:20.906473 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:20.906435    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:21.010031 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.009995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:21.138397 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.138351    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"]
Jul 13 22:11:21.138667 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.138637    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6" podUID="79aab0ac-1575-4ac6-9185-19ad53b37acf" containerName="maas-api" containerID="cri-o://d11a9a150f7d5a32d3097a39a2bb5550932e9e56383ded607e12ab9fad51d9da" gracePeriod=30
Jul 13 22:11:21.167197 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.167127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:21.313527 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.313501    2582 generic.go:356] "Generic (PLEG): container finished" podID="79aab0ac-1575-4ac6-9185-19ad53b37acf" containerID="d11a9a150f7d5a32d3097a39a2bb5550932e9e56383ded607e12ab9fad51d9da" exitCode=0
Jul 13 22:11:21.313642 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.313574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6" event={"ID":"79aab0ac-1575-4ac6-9185-19ad53b37acf","Type":"ContainerDied","Data":"d11a9a150f7d5a32d3097a39a2bb5550932e9e56383ded607e12ab9fad51d9da"}
Jul 13 22:11:21.384805 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.384781    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:11:21.535860 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.535820    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-996zr\" (UniqueName: \"kubernetes.io/projected/79aab0ac-1575-4ac6-9185-19ad53b37acf-kube-api-access-996zr\") pod \"79aab0ac-1575-4ac6-9185-19ad53b37acf\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") "
Jul 13 22:11:21.536042 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.535929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79aab0ac-1575-4ac6-9185-19ad53b37acf-openshift-service-ca\") pod \"79aab0ac-1575-4ac6-9185-19ad53b37acf\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") "
Jul 13 22:11:21.536042 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.535951    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79aab0ac-1575-4ac6-9185-19ad53b37acf-maas-api-tls\") pod \"79aab0ac-1575-4ac6-9185-19ad53b37acf\" (UID: \"79aab0ac-1575-4ac6-9185-19ad53b37acf\") "
Jul 13 22:11:21.536276 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.536251    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79aab0ac-1575-4ac6-9185-19ad53b37acf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79aab0ac-1575-4ac6-9185-19ad53b37acf" (UID: "79aab0ac-1575-4ac6-9185-19ad53b37acf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:11:21.537908 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.537885    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79aab0ac-1575-4ac6-9185-19ad53b37acf-kube-api-access-996zr" (OuterVolumeSpecName: "kube-api-access-996zr") pod "79aab0ac-1575-4ac6-9185-19ad53b37acf" (UID: "79aab0ac-1575-4ac6-9185-19ad53b37acf"). InnerVolumeSpecName "kube-api-access-996zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:11:21.537978 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.537910    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79aab0ac-1575-4ac6-9185-19ad53b37acf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79aab0ac-1575-4ac6-9185-19ad53b37acf" (UID: "79aab0ac-1575-4ac6-9185-19ad53b37acf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:11:21.637416 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.637388    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-996zr\" (UniqueName: \"kubernetes.io/projected/79aab0ac-1575-4ac6-9185-19ad53b37acf-kube-api-access-996zr\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:11:21.637416 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.637412    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79aab0ac-1575-4ac6-9185-19ad53b37acf-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:11:21.637549 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:21.637423    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79aab0ac-1575-4ac6-9185-19ad53b37acf-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:11:22.320135 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:22.320097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6" event={"ID":"79aab0ac-1575-4ac6-9185-19ad53b37acf","Type":"ContainerDied","Data":"66d64d6bdfcede252c65e4c9d5f9b89d70a288ca4b9d7cabe892515405ff58a7"}
Jul 13 22:11:22.320135 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:22.320126    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"
Jul 13 22:11:22.320610 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:22.320196    2582 scope.go:117] "RemoveContainer" containerID="d11a9a150f7d5a32d3097a39a2bb5550932e9e56383ded607e12ab9fad51d9da"
Jul 13 22:11:22.342464 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:22.342442    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"]
Jul 13 22:11:22.346327 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:22.346308    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-785879-8588dfb954-xn2s6"]
Jul 13 22:11:23.362670 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:23.362633    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79aab0ac-1575-4ac6-9185-19ad53b37acf" path="/var/lib/kubelet/pods/79aab0ac-1575-4ac6-9185-19ad53b37acf/volumes"
Jul 13 22:11:24.946331 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:24.946297    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:11:25.469880 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:25.469841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:31.206169 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:31.206112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:31.263883 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:31.263845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:36.160673 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.160637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:36.396774 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.396738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:36.516655 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.516621    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh"]
Jul 13 22:11:36.516963 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.516951    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="912ba5d2-ec45-4a89-a8d3-7159bf7f59e8" containerName="revoke-keys"
Jul 13 22:11:36.517008 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.516964    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="912ba5d2-ec45-4a89-a8d3-7159bf7f59e8" containerName="revoke-keys"
Jul 13 22:11:36.517008 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.516976    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79aab0ac-1575-4ac6-9185-19ad53b37acf" containerName="maas-api"
Jul 13 22:11:36.517008 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.516982    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="79aab0ac-1575-4ac6-9185-19ad53b37acf" containerName="maas-api"
Jul 13 22:11:36.517107 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.517044    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="912ba5d2-ec45-4a89-a8d3-7159bf7f59e8" containerName="revoke-keys"
Jul 13 22:11:36.517107 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.517055    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="79aab0ac-1575-4ac6-9185-19ad53b37acf" containerName="maas-api"
Jul 13 22:11:36.520353 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.520336    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh"
Jul 13 22:11:36.522919 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.522898    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:11:36.526904 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.526881    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh"]
Jul 13 22:11:36.550628 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.550595    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6083269a-308c-4dfc-96c0-d28a9639a7d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh\" (UID: \"6083269a-308c-4dfc-96c0-d28a9639a7d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh"
Jul 13 22:11:36.652120 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.652081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6083269a-308c-4dfc-96c0-d28a9639a7d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh\" (UID: \"6083269a-308c-4dfc-96c0-d28a9639a7d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh"
Jul 13 22:11:36.652721 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.652702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6083269a-308c-4dfc-96c0-d28a9639a7d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh\" (UID: \"6083269a-308c-4dfc-96c0-d28a9639a7d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh"
Jul 13 22:11:36.832384 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.832302    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh"
Jul 13 22:11:36.965710 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:36.965683    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh"]
Jul 13 22:11:36.966798 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:11:36.966771    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6083269a_308c_4dfc_96c0_d28a9639a7d0.slice/crio-1ea061ac14f6db1b5190d5000adaa483ed270a32d252ea7f45c1bfd25f9fba3f WatchSource:0}: Error finding container 1ea061ac14f6db1b5190d5000adaa483ed270a32d252ea7f45c1bfd25f9fba3f: Status 404 returned error can't find the container with id 1ea061ac14f6db1b5190d5000adaa483ed270a32d252ea7f45c1bfd25f9fba3f
Jul 13 22:11:37.028370 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:37.028343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:37.129782 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:37.129752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:37.399989 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:37.399905    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh" event={"ID":"6083269a-308c-4dfc-96c0-d28a9639a7d0","Type":"ContainerStarted","Data":"322f06d606b8d4217eca5f3a9c2f7f6c80ea5146777e4148b3b878f2e6cf9d1c"}
Jul 13 22:11:37.399989 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:37.399953    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh" event={"ID":"6083269a-308c-4dfc-96c0-d28a9639a7d0","Type":"ContainerStarted","Data":"1ea061ac14f6db1b5190d5000adaa483ed270a32d252ea7f45c1bfd25f9fba3f"}
Jul 13 22:11:37.436398 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:37.436353    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh" podStartSLOduration=1.436337875 podStartE2EDuration="1.436337875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:11:37.433895554 +0000 UTC m=+2538.590289394" watchObservedRunningTime="2026-07-13 22:11:37.436337875 +0000 UTC m=+2538.592731716"
Jul 13 22:11:38.406853 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:38.406821    2582 generic.go:356] "Generic (PLEG): container finished" podID="6083269a-308c-4dfc-96c0-d28a9639a7d0" containerID="322f06d606b8d4217eca5f3a9c2f7f6c80ea5146777e4148b3b878f2e6cf9d1c" exitCode=0
Jul 13 22:11:38.407245 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:38.406899    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh" event={"ID":"6083269a-308c-4dfc-96c0-d28a9639a7d0","Type":"ContainerDied","Data":"322f06d606b8d4217eca5f3a9c2f7f6c80ea5146777e4148b3b878f2e6cf9d1c"}
Jul 13 22:11:39.546493 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:39.546469    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh"
Jul 13 22:11:39.575813 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:39.575787    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6083269a-308c-4dfc-96c0-d28a9639a7d0-maas-api-service-ca\") pod \"6083269a-308c-4dfc-96c0-d28a9639a7d0\" (UID: \"6083269a-308c-4dfc-96c0-d28a9639a7d0\") "
Jul 13 22:11:39.576141 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:39.576117    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6083269a-308c-4dfc-96c0-d28a9639a7d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6083269a-308c-4dfc-96c0-d28a9639a7d0" (UID: "6083269a-308c-4dfc-96c0-d28a9639a7d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:11:39.676641 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:39.676568    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6083269a-308c-4dfc-96c0-d28a9639a7d0-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:11:40.419547 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:40.419511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh" event={"ID":"6083269a-308c-4dfc-96c0-d28a9639a7d0","Type":"ContainerDied","Data":"1ea061ac14f6db1b5190d5000adaa483ed270a32d252ea7f45c1bfd25f9fba3f"}
Jul 13 22:11:40.419547 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:40.419533    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh"
Jul 13 22:11:40.419796 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:40.419569    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ea061ac14f6db1b5190d5000adaa483ed270a32d252ea7f45c1bfd25f9fba3f"
Jul 13 22:11:46.919960 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:46.919927    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"]
Jul 13 22:11:46.920434 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:46.920139    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k" podUID="70e49acc-65a4-4cbe-801b-fc050e2407c3" containerName="maas-api" containerID="cri-o://c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a" gracePeriod=30
Jul 13 22:11:46.946857 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:46.946828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:47.175080 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.175001    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:11:47.240952 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.240921    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6js5s\" (UniqueName: \"kubernetes.io/projected/70e49acc-65a4-4cbe-801b-fc050e2407c3-kube-api-access-6js5s\") pod \"70e49acc-65a4-4cbe-801b-fc050e2407c3\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") "
Jul 13 22:11:47.241088 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.240969    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e49acc-65a4-4cbe-801b-fc050e2407c3-maas-api-tls\") pod \"70e49acc-65a4-4cbe-801b-fc050e2407c3\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") "
Jul 13 22:11:47.241088 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.241018    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e49acc-65a4-4cbe-801b-fc050e2407c3-openshift-service-ca\") pod \"70e49acc-65a4-4cbe-801b-fc050e2407c3\" (UID: \"70e49acc-65a4-4cbe-801b-fc050e2407c3\") "
Jul 13 22:11:47.241335 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.241314    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e49acc-65a4-4cbe-801b-fc050e2407c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70e49acc-65a4-4cbe-801b-fc050e2407c3" (UID: "70e49acc-65a4-4cbe-801b-fc050e2407c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:11:47.243063 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.243035    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e49acc-65a4-4cbe-801b-fc050e2407c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70e49acc-65a4-4cbe-801b-fc050e2407c3" (UID: "70e49acc-65a4-4cbe-801b-fc050e2407c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:11:47.243177 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.243079    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e49acc-65a4-4cbe-801b-fc050e2407c3-kube-api-access-6js5s" (OuterVolumeSpecName: "kube-api-access-6js5s") pod "70e49acc-65a4-4cbe-801b-fc050e2407c3" (UID: "70e49acc-65a4-4cbe-801b-fc050e2407c3"). InnerVolumeSpecName "kube-api-access-6js5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:11:47.342273 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.342239    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6js5s\" (UniqueName: \"kubernetes.io/projected/70e49acc-65a4-4cbe-801b-fc050e2407c3-kube-api-access-6js5s\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:11:47.342273 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.342264    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e49acc-65a4-4cbe-801b-fc050e2407c3-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:11:47.342273 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.342275    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e49acc-65a4-4cbe-801b-fc050e2407c3-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:11:47.455166 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.455080    2582 generic.go:356] "Generic (PLEG): container finished" podID="70e49acc-65a4-4cbe-801b-fc050e2407c3" containerID="c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a" exitCode=0
Jul 13 22:11:47.455166 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.455128    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k" event={"ID":"70e49acc-65a4-4cbe-801b-fc050e2407c3","Type":"ContainerDied","Data":"c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a"}
Jul 13 22:11:47.455166 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.455138    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"
Jul 13 22:11:47.455369 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.455190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k" event={"ID":"70e49acc-65a4-4cbe-801b-fc050e2407c3","Type":"ContainerDied","Data":"8aa291c183b563ca22a7f2126782cf2861f491dba3ddd1d69724bc3e68bf79be"}
Jul 13 22:11:47.455369 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.455214    2582 scope.go:117] "RemoveContainer" containerID="c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a"
Jul 13 22:11:47.463985 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.463963    2582 scope.go:117] "RemoveContainer" containerID="c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a"
Jul 13 22:11:47.464294 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:11:47.464275    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a\": container with ID starting with c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a not found: ID does not exist" containerID="c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a"
Jul 13 22:11:47.464354 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.464306    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a"} err="failed to get container status \"c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a\": rpc error: code = NotFound desc = could not find container \"c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a\": container with ID starting with c8bb53c2c0fa296ba0d1193c58025932c38776458a5af678cd3a1d13c7d11f2a not found: ID does not exist"
Jul 13 22:11:47.479510 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.479485    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"]
Jul 13 22:11:47.484508 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:47.484488    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9dff5a-5f878466d-kzd7k"]
Jul 13 22:11:49.364730 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:49.364699    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70e49acc-65a4-4cbe-801b-fc050e2407c3" path="/var/lib/kubelet/pods/70e49acc-65a4-4cbe-801b-fc050e2407c3/volumes"
Jul 13 22:11:50.158872 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:50.155816    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:11:51.596497 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:51.596463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:57.429745 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:57.429708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:11:57.486909 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:11:57.486876    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:02.274043 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:02.273670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:02.503929 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:02.503886    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:02.839406 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:02.839367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:03.151631 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:03.151538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:03.414616 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:03.414524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:06.273430 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.273378    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:06.372161 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.372118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:06.463752 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.463722    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"]
Jul 13 22:12:06.464052 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.464040    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e49acc-65a4-4cbe-801b-fc050e2407c3" containerName="maas-api"
Jul 13 22:12:06.464104 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.464053    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e49acc-65a4-4cbe-801b-fc050e2407c3" containerName="maas-api"
Jul 13 22:12:06.464104 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.464073    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6083269a-308c-4dfc-96c0-d28a9639a7d0" containerName="revoke-keys"
Jul 13 22:12:06.464104 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.464079    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6083269a-308c-4dfc-96c0-d28a9639a7d0" containerName="revoke-keys"
Jul 13 22:12:06.464236 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.464137    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="70e49acc-65a4-4cbe-801b-fc050e2407c3" containerName="maas-api"
Jul 13 22:12:06.464236 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.464146    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6083269a-308c-4dfc-96c0-d28a9639a7d0" containerName="revoke-keys"
Jul 13 22:12:06.466784 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.466767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:06.469361 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.469335    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-f89343-serving-cert\""
Jul 13 22:12:06.469616 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.469601    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kfxsg\""
Jul 13 22:12:06.469687 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.469624    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 22:12:06.470906 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.470884    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 22:12:06.477764 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.477734    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"]
Jul 13 22:12:06.516171 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.516115    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:06.616738 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.616647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac79734-8ba9-4de9-adda-be140fc8f3f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:06.616738 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.616685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9fcb\" (UniqueName: \"kubernetes.io/projected/2ac79734-8ba9-4de9-adda-be140fc8f3f4-kube-api-access-x9fcb\") pod \"maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:06.616936 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.616806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac79734-8ba9-4de9-adda-be140fc8f3f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:06.717673 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.717637    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac79734-8ba9-4de9-adda-be140fc8f3f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:06.717839 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.717706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac79734-8ba9-4de9-adda-be140fc8f3f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:06.717839 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.717735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9fcb\" (UniqueName: \"kubernetes.io/projected/2ac79734-8ba9-4de9-adda-be140fc8f3f4-kube-api-access-x9fcb\") pod \"maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:06.717839 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:12:06.717781    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-serving-cert: secret "maas-api-e2e-shared-a-f89343-serving-cert" not found
Jul 13 22:12:06.717947 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:12:06.717858    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ac79734-8ba9-4de9-adda-be140fc8f3f4-maas-api-tls podName:2ac79734-8ba9-4de9-adda-be140fc8f3f4 nodeName:}" failed. No retries permitted until 2026-07-13 22:12:07.217840679 +0000 UTC m=+2568.374234497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ac79734-8ba9-4de9-adda-be140fc8f3f4-maas-api-tls") pod "maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk" (UID: "2ac79734-8ba9-4de9-adda-be140fc8f3f4") : secret "maas-api-e2e-shared-a-f89343-serving-cert" not found
Jul 13 22:12:06.718463 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.718446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac79734-8ba9-4de9-adda-be140fc8f3f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:06.726572 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.726553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9fcb\" (UniqueName: \"kubernetes.io/projected/2ac79734-8ba9-4de9-adda-be140fc8f3f4-kube-api-access-x9fcb\") pod \"maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:06.967299 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:06.967205    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:07.141855 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:07.141820    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:07.223497 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:07.223424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac79734-8ba9-4de9-adda-be140fc8f3f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:07.225839 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:07.225813    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac79734-8ba9-4de9-adda-be140fc8f3f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:07.243861 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:07.243829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:07.380320 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:07.380289    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:07.518295 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:07.518272    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"]
Jul 13 22:12:07.520356 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:12:07.520330    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac79734_8ba9_4de9_adda_be140fc8f3f4.slice/crio-6e26d7ea591a6b3f4b9afba64da7ce88e29522990ed710b31c62427253e3bc1c WatchSource:0}: Error finding container 6e26d7ea591a6b3f4b9afba64da7ce88e29522990ed710b31c62427253e3bc1c: Status 404 returned error can't find the container with id 6e26d7ea591a6b3f4b9afba64da7ce88e29522990ed710b31c62427253e3bc1c
Jul 13 22:12:07.554954 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:07.554921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk" event={"ID":"2ac79734-8ba9-4de9-adda-be140fc8f3f4","Type":"ContainerStarted","Data":"6e26d7ea591a6b3f4b9afba64da7ce88e29522990ed710b31c62427253e3bc1c"}
Jul 13 22:12:08.561518 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:08.561484    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk" event={"ID":"2ac79734-8ba9-4de9-adda-be140fc8f3f4","Type":"ContainerStarted","Data":"6a4781d2756ca732822c60c8a088915b6907a75eba451c264fec0e632cf4834d"}
Jul 13 22:12:08.561893 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:08.561610    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:08.580103 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:08.580055    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk" podStartSLOduration=2.227861751 podStartE2EDuration="2.580041619s" totalImagesPullingTime="352.179868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:12:08.578039914 +0000 UTC m=+2569.734433779" watchObservedRunningTime="2026-07-13 22:12:08.580041619 +0000 UTC m=+2569.736435460"
Jul 13 22:12:10.772413 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:10.772382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:11.045265 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:11.045177    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:12:11.107735 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:11.107695    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:11.419194 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:11.419090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:12.018930 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:12.018893    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:14.282513 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.282475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:14.366235 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.366202    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:14.386627 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.386598    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"]
Jul 13 22:12:14.391519 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.391500    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:14.394184 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.394142    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-d8932b-serving-cert\""
Jul 13 22:12:14.396971 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.396946    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"]
Jul 13 22:12:14.466164 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.466127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:14.482889 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.482863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:14.482985 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.482917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l72ql\" (UniqueName: \"kubernetes.io/projected/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-kube-api-access-l72ql\") pod \"maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:14.482985 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.482966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:14.571748 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.571691    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:12:14.583731 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.583705    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l72ql\" (UniqueName: \"kubernetes.io/projected/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-kube-api-access-l72ql\") pod \"maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:14.583836 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.583736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:14.583884 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.583840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:14.583981 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:12:14.583963    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-serving-cert: secret "maas-api-e2e-shared-b-d8932b-serving-cert" not found
Jul 13 22:12:14.584058 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:12:14.584045    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-maas-api-tls podName:2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7 nodeName:}" failed. No retries permitted until 2026-07-13 22:12:15.084025779 +0000 UTC m=+2576.240419600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-maas-api-tls") pod "maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd" (UID: "2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7") : secret "maas-api-e2e-shared-b-d8932b-serving-cert" not found
Jul 13 22:12:14.584483 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.584467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:14.597398 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.597376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l72ql\" (UniqueName: \"kubernetes.io/projected/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-kube-api-access-l72ql\") pod \"maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:14.703303 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:14.703274    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:15.087450 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:15.087411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:15.089854 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:15.089832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:15.099306 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:15.099282    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:15.224619 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:15.224592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:15.304114 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:15.304087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:15.445573 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:15.445539    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"]
Jul 13 22:12:15.447751 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:12:15.447723    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6b0fb6_7897_4ef2_ab20_c72d96b8d1a7.slice/crio-0e6221535d406d0b073b592608415d2bd5dedfeaac6f62dde63ca71adcc0a591 WatchSource:0}: Error finding container 0e6221535d406d0b073b592608415d2bd5dedfeaac6f62dde63ca71adcc0a591: Status 404 returned error can't find the container with id 0e6221535d406d0b073b592608415d2bd5dedfeaac6f62dde63ca71adcc0a591
Jul 13 22:12:15.598952 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:15.598918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd" event={"ID":"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7","Type":"ContainerStarted","Data":"0e6221535d406d0b073b592608415d2bd5dedfeaac6f62dde63ca71adcc0a591"}
Jul 13 22:12:16.606365 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:16.606332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd" event={"ID":"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7","Type":"ContainerStarted","Data":"3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf"}
Jul 13 22:12:16.606771 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:16.606438    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:16.635929 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:16.635878    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd" podStartSLOduration=2.188724405 podStartE2EDuration="2.635864523s" totalImagesPullingTime="447.140118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:12:16.634384495 +0000 UTC m=+2577.790778337" watchObservedRunningTime="2026-07-13 22:12:16.635864523 +0000 UTC m=+2577.792258364"
Jul 13 22:12:18.788120 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:18.788080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:19.953246 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:19.951691    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:12:22.616145 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:22.616115    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:12:27.981877 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:27.981829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:28.078511 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:28.078470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:33.568047 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:33.568011    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:35.924325 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:35.924293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:36.023561 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:36.023530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:37.054550 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:37.054512    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:12:50.631085 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.631004    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"]
Jul 13 22:12:50.636011 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.635986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.638635 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.638612    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d8932b\"/\"kube-root-ca.crt\""
Jul 13 22:12:50.638745 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.638664    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d8932b\"/\"openshift-service-ca.crt\""
Jul 13 22:12:50.639731 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.639710    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d8932b\"/\"default-dockercfg-hdwjt\""
Jul 13 22:12:50.639847 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.639750    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d8932b\"/\"auth-test-model-d8932b-kserve-self-signed-certs\""
Jul 13 22:12:50.644570 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.644547    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"]
Jul 13 22:12:50.657706 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.657677    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:12:50.776938 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.776907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tmp-dir\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.776938 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.776939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-model-cache\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.777135 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.776973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-home\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.777135 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.777028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tls-certs\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.777135 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.777086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-dshm\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.777135 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.777105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnp76\" (UniqueName: \"kubernetes.io/projected/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-kube-api-access-hnp76\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.878192 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.878136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tls-certs\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.878357 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.878248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-dshm\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.878357 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.878283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnp76\" (UniqueName: \"kubernetes.io/projected/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-kube-api-access-hnp76\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.878357 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.878316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tmp-dir\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.878522 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.878371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-model-cache\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.878522 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.878454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-home\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.878775 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.878742    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-model-cache\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.878910 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.878817    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-home\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.878910 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.878863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tmp-dir\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.880488 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.880461    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-dshm\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.880715 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.880699    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tls-certs\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.886730 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.886677    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnp76\" (UniqueName: \"kubernetes.io/projected/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-kube-api-access-hnp76\") pod \"auth-test-model-d8932b-kserve-6d67ff5855-k9qlv\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") " pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:50.952753 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:50.952731    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:12:51.081004 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:51.080487    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"]
Jul 13 22:12:51.082881 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:12:51.082855    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb87262d7_e35b_42a4_8595_b1bbfbbd4d68.slice/crio-88d4619754f7532e80d5a3beb4d43b69edc1d3153f9b65007ec8e8b88677c25c WatchSource:0}: Error finding container 88d4619754f7532e80d5a3beb4d43b69edc1d3153f9b65007ec8e8b88677c25c: Status 404 returned error can't find the container with id 88d4619754f7532e80d5a3beb4d43b69edc1d3153f9b65007ec8e8b88677c25c
Jul 13 22:12:51.780254 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:51.780213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv" event={"ID":"b87262d7-e35b-42a4-8595-b1bbfbbd4d68","Type":"ContainerStarted","Data":"c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8"}
Jul 13 22:12:51.780254 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:51.780259    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv" event={"ID":"b87262d7-e35b-42a4-8595-b1bbfbbd4d68","Type":"ContainerStarted","Data":"88d4619754f7532e80d5a3beb4d43b69edc1d3153f9b65007ec8e8b88677c25c"}
Jul 13 22:12:51.799027 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:51.798976    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv" podStartSLOduration=1.798961137 podStartE2EDuration="1.798961137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:12:51.797485679 +0000 UTC m=+2612.953879523" watchObservedRunningTime="2026-07-13 22:12:51.798961137 +0000 UTC m=+2612.955354978"
Jul 13 22:12:54.447588 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:12:54.447546    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:13:00.952871 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:13:00.952832    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:13:00.953346 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:13:00.953091    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:13:00.967678 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:13:00.967655    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:13:01.839964 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:13:01.839934    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:13:10.455654 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:13:10.455614    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:13:29.054749 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:13:29.054713    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:13:44.657461 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:13:44.657423    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:14:02.554896 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:02.554858    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:14:19.419492 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:19.419415    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 22:14:19.424389 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:19.424367    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 22:14:19.427703 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:19.427670    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 22:14:19.432279 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:19.432263    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 22:14:21.655753 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:21.655718    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:14:39.145660 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:39.145627    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:14:52.769692 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:52.769658    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:14:53.551050 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:53.551017    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"]
Jul 13 22:14:54.225041 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:54.225008    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:14:56.557131 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:14:56.557099    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:15:00.138827 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.138792    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7"]
Jul 13 22:15:00.142113 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.142096    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2"]
Jul 13 22:15:00.142271 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.142249    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7"
Jul 13 22:15:00.144706 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.144685    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:15:00.145169 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.145137    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql"]
Jul 13 22:15:00.145295 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.145278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2"
Jul 13 22:15:00.148180 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.148145    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql"
Jul 13 22:15:00.150540 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.150069    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7"]
Jul 13 22:15:00.153590 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.153569    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2"]
Jul 13 22:15:00.155737 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.155715    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7"
Jul 13 22:15:00.164491 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.164470    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql"]
Jul 13 22:15:00.165832 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.165809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2"
Jul 13 22:15:00.174849 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.174829    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql"
Jul 13 22:15:00.318581 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.318553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7"]
Jul 13 22:15:00.321517 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:15:00.321488    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc81a9be_d077_43ec_8e18_f33cb60ab425.slice/crio-fe7cb3cae14ec2968e190fc84aaaa81e3535d55013a80352d595fa2b6e0d4ece WatchSource:0}: Error finding container fe7cb3cae14ec2968e190fc84aaaa81e3535d55013a80352d595fa2b6e0d4ece: Status 404 returned error can't find the container with id fe7cb3cae14ec2968e190fc84aaaa81e3535d55013a80352d595fa2b6e0d4ece
Jul 13 22:15:00.362301 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.362277    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql"]
Jul 13 22:15:00.363885 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:15:00.363864    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf5c175_46c5_4032_a341_09396406df71.slice/crio-7b10a9b9fd5ddc3c46708a1762c3141070595dea3dc86df3fc07fc8538dcf001 WatchSource:0}: Error finding container 7b10a9b9fd5ddc3c46708a1762c3141070595dea3dc86df3fc07fc8538dcf001: Status 404 returned error can't find the container with id 7b10a9b9fd5ddc3c46708a1762c3141070595dea3dc86df3fc07fc8538dcf001
Jul 13 22:15:00.412919 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.412894    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql" event={"ID":"2bf5c175-46c5-4032-a341-09396406df71","Type":"ContainerStarted","Data":"7b10a9b9fd5ddc3c46708a1762c3141070595dea3dc86df3fc07fc8538dcf001"}
Jul 13 22:15:00.413954 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.413932    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7" event={"ID":"cc81a9be-d077-43ec-8e18-f33cb60ab425","Type":"ContainerStarted","Data":"fe7cb3cae14ec2968e190fc84aaaa81e3535d55013a80352d595fa2b6e0d4ece"}
Jul 13 22:15:00.543320 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:00.543299    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2"]
Jul 13 22:15:00.605964 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:15:00.605923    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989f6e36_68cb_455c_89de_cada265e3a3c.slice/crio-cc50a7122ba191d98a272a8ac11af4790140464143be2e8ba675adf013fc5657 WatchSource:0}: Error finding container cc50a7122ba191d98a272a8ac11af4790140464143be2e8ba675adf013fc5657: Status 404 returned error can't find the container with id cc50a7122ba191d98a272a8ac11af4790140464143be2e8ba675adf013fc5657
Jul 13 22:15:01.420506 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:01.420463    2582 generic.go:356] "Generic (PLEG): container finished" podID="2bf5c175-46c5-4032-a341-09396406df71" containerID="621776a916a6f5535a01deb70926140caa1170fabdb91ca608c7d80329eb3a1f" exitCode=0
Jul 13 22:15:01.420954 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:01.420551    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql" event={"ID":"2bf5c175-46c5-4032-a341-09396406df71","Type":"ContainerDied","Data":"621776a916a6f5535a01deb70926140caa1170fabdb91ca608c7d80329eb3a1f"}
Jul 13 22:15:01.422202 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:01.422176    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2" event={"ID":"989f6e36-68cb-455c-89de-cada265e3a3c","Type":"ContainerStarted","Data":"6740dd31a4fcc00a9ded657314b0951ba23b8462acf68295b9e1d932584bf45a"}
Jul 13 22:15:01.422341 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:01.422223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2" event={"ID":"989f6e36-68cb-455c-89de-cada265e3a3c","Type":"ContainerStarted","Data":"cc50a7122ba191d98a272a8ac11af4790140464143be2e8ba675adf013fc5657"}
Jul 13 22:15:01.423642 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:01.423614    2582 generic.go:356] "Generic (PLEG): container finished" podID="cc81a9be-d077-43ec-8e18-f33cb60ab425" containerID="5931bd91e32882e74153d73b1d12d4fee1da891e19ebaafae6920840585a9f43" exitCode=0
Jul 13 22:15:01.423751 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:01.423644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7" event={"ID":"cc81a9be-d077-43ec-8e18-f33cb60ab425","Type":"ContainerDied","Data":"5931bd91e32882e74153d73b1d12d4fee1da891e19ebaafae6920840585a9f43"}
Jul 13 22:15:01.463201 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:01.463079    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2" podStartSLOduration=1.463061683 podStartE2EDuration="1.463061683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:15:01.461639362 +0000 UTC m=+2742.618033202" watchObservedRunningTime="2026-07-13 22:15:01.463061683 +0000 UTC m=+2742.619455522"
Jul 13 22:15:02.430033 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:02.429995    2582 generic.go:356] "Generic (PLEG): container finished" podID="989f6e36-68cb-455c-89de-cada265e3a3c" containerID="6740dd31a4fcc00a9ded657314b0951ba23b8462acf68295b9e1d932584bf45a" exitCode=0
Jul 13 22:15:02.430521 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:02.430079    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2" event={"ID":"989f6e36-68cb-455c-89de-cada265e3a3c","Type":"ContainerDied","Data":"6740dd31a4fcc00a9ded657314b0951ba23b8462acf68295b9e1d932584bf45a"}
Jul 13 22:15:02.572243 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:02.572215    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7"
Jul 13 22:15:02.598412 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:02.598385    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql"
Jul 13 22:15:02.656734 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:02.656703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:15:03.436855 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:03.436830    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql"
Jul 13 22:15:03.437264 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:03.436872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql" event={"ID":"2bf5c175-46c5-4032-a341-09396406df71","Type":"ContainerDied","Data":"7b10a9b9fd5ddc3c46708a1762c3141070595dea3dc86df3fc07fc8538dcf001"}
Jul 13 22:15:03.437264 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:03.436924    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b10a9b9fd5ddc3c46708a1762c3141070595dea3dc86df3fc07fc8538dcf001"
Jul 13 22:15:03.438181 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:03.438160    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7"
Jul 13 22:15:03.438308 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:03.438198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7" event={"ID":"cc81a9be-d077-43ec-8e18-f33cb60ab425","Type":"ContainerDied","Data":"fe7cb3cae14ec2968e190fc84aaaa81e3535d55013a80352d595fa2b6e0d4ece"}
Jul 13 22:15:03.438308 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:03.438235    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe7cb3cae14ec2968e190fc84aaaa81e3535d55013a80352d595fa2b6e0d4ece"
Jul 13 22:15:03.574922 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:03.574897    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2"
Jul 13 22:15:04.445346 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:04.445315    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2"
Jul 13 22:15:04.445346 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:04.445345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-6jqd2" event={"ID":"989f6e36-68cb-455c-89de-cada265e3a3c","Type":"ContainerDied","Data":"cc50a7122ba191d98a272a8ac11af4790140464143be2e8ba675adf013fc5657"}
Jul 13 22:15:04.445760 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:04.445388    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc50a7122ba191d98a272a8ac11af4790140464143be2e8ba675adf013fc5657"
Jul 13 22:15:08.571005 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.570964    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv" podUID="b87262d7-e35b-42a4-8595-b1bbfbbd4d68" containerName="main" containerID="cri-o://c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8" gracePeriod=45
Jul 13 22:15:08.821671 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.821614    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:15:08.854073 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854043    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tmp-dir\") pod \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") "
Jul 13 22:15:08.854234 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854090    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnp76\" (UniqueName: \"kubernetes.io/projected/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-kube-api-access-hnp76\") pod \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") "
Jul 13 22:15:08.854234 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854120    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-home\") pod \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") "
Jul 13 22:15:08.854234 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854162    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-dshm\") pod \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") "
Jul 13 22:15:08.854234 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854210    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-model-cache\") pod \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") "
Jul 13 22:15:08.854452 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854237    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tls-certs\") pod \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\" (UID: \"b87262d7-e35b-42a4-8595-b1bbfbbd4d68\") "
Jul 13 22:15:08.854452 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854353    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b87262d7-e35b-42a4-8595-b1bbfbbd4d68" (UID: "b87262d7-e35b-42a4-8595-b1bbfbbd4d68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:15:08.854554 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854457    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-home" (OuterVolumeSpecName: "home") pod "b87262d7-e35b-42a4-8595-b1bbfbbd4d68" (UID: "b87262d7-e35b-42a4-8595-b1bbfbbd4d68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:15:08.854612 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854575    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-model-cache" (OuterVolumeSpecName: "model-cache") pod "b87262d7-e35b-42a4-8595-b1bbfbbd4d68" (UID: "b87262d7-e35b-42a4-8595-b1bbfbbd4d68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:15:08.854612 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854602    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tmp-dir\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:15:08.854710 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.854615    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-home\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:15:08.856401 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.856371    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-dshm" (OuterVolumeSpecName: "dshm") pod "b87262d7-e35b-42a4-8595-b1bbfbbd4d68" (UID: "b87262d7-e35b-42a4-8595-b1bbfbbd4d68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:15:08.856401 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.856380    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-kube-api-access-hnp76" (OuterVolumeSpecName: "kube-api-access-hnp76") pod "b87262d7-e35b-42a4-8595-b1bbfbbd4d68" (UID: "b87262d7-e35b-42a4-8595-b1bbfbbd4d68"). InnerVolumeSpecName "kube-api-access-hnp76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:15:08.856539 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.856426    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b87262d7-e35b-42a4-8595-b1bbfbbd4d68" (UID: "b87262d7-e35b-42a4-8595-b1bbfbbd4d68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:15:08.955707 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.955676    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnp76\" (UniqueName: \"kubernetes.io/projected/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-kube-api-access-hnp76\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:15:08.955707 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.955704    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-dshm\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:15:08.955888 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.955714    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-model-cache\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:15:08.955888 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:08.955725    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b87262d7-e35b-42a4-8595-b1bbfbbd4d68-tls-certs\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:15:09.470962 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:09.470933    2582 generic.go:356] "Generic (PLEG): container finished" podID="b87262d7-e35b-42a4-8595-b1bbfbbd4d68" containerID="c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8" exitCode=0
Jul 13 22:15:09.471112 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:09.470978    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"
Jul 13 22:15:09.471112 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:09.471016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv" event={"ID":"b87262d7-e35b-42a4-8595-b1bbfbbd4d68","Type":"ContainerDied","Data":"c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8"}
Jul 13 22:15:09.471112 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:09.471058    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv" event={"ID":"b87262d7-e35b-42a4-8595-b1bbfbbd4d68","Type":"ContainerDied","Data":"88d4619754f7532e80d5a3beb4d43b69edc1d3153f9b65007ec8e8b88677c25c"}
Jul 13 22:15:09.471112 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:09.471082    2582 scope.go:117] "RemoveContainer" containerID="c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8"
Jul 13 22:15:09.479841 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:09.479824    2582 scope.go:117] "RemoveContainer" containerID="c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8"
Jul 13 22:15:09.480095 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:15:09.480077    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8\": container with ID starting with c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8 not found: ID does not exist" containerID="c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8"
Jul 13 22:15:09.480173 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:09.480103    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8"} err="failed to get container status \"c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8\": rpc error: code = NotFound desc = could not find container \"c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8\": container with ID starting with c5c610039d8909c8962911989324f91d075697e8ec960d26a67ae18e7c0076c8 not found: ID does not exist"
Jul 13 22:15:09.489076 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:09.489055    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"]
Jul 13 22:15:09.491101 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:09.491081    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/auth-test-model-d8932b-kserve-6d67ff5855-k9qlv"]
Jul 13 22:15:11.364138 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:11.364104    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b87262d7-e35b-42a4-8595-b1bbfbbd4d68" path="/var/lib/kubelet/pods/b87262d7-e35b-42a4-8595-b1bbfbbd4d68/volumes"
Jul 13 22:15:13.247480 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:13.247426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:15:13.298699 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:13.298666    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:15:15.551846 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:15.551817    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:15:38.865490 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:38.865457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:15:43.152807 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:43.152725    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:15:58.649037 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:15:58.649001    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:16:05.700552 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:16:05.700510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:16:05.800815 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:16:05.800775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:16:18.548038 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:16:18.547997    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:16:25.031864 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:16:25.031828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:16:25.927927 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:16:25.927891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:16:36.946008 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:16:36.945973    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:16:40.849723 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:16:40.849689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:16:54.045444 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:16:54.045406    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:16:56.742894 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:16:56.742843    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:16:57.744612 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:16:57.744576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:17:09.237772 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.237738    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"]
Jul 13 22:17:09.238128 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238110    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc81a9be-d077-43ec-8e18-f33cb60ab425" containerName="cleanup"
Jul 13 22:17:09.238128 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238122    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc81a9be-d077-43ec-8e18-f33cb60ab425" containerName="cleanup"
Jul 13 22:17:09.238229 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238137    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87262d7-e35b-42a4-8595-b1bbfbbd4d68" containerName="main"
Jul 13 22:17:09.238229 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238143    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87262d7-e35b-42a4-8595-b1bbfbbd4d68" containerName="main"
Jul 13 22:17:09.238229 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238178    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf5c175-46c5-4032-a341-09396406df71" containerName="cleanup"
Jul 13 22:17:09.238229 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238184    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf5c175-46c5-4032-a341-09396406df71" containerName="cleanup"
Jul 13 22:17:09.238229 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238198    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989f6e36-68cb-455c-89de-cada265e3a3c" containerName="cleanup"
Jul 13 22:17:09.238229 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238203    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="989f6e36-68cb-455c-89de-cada265e3a3c" containerName="cleanup"
Jul 13 22:17:09.238415 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238257    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc81a9be-d077-43ec-8e18-f33cb60ab425" containerName="cleanup"
Jul 13 22:17:09.238415 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238267    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf5c175-46c5-4032-a341-09396406df71" containerName="cleanup"
Jul 13 22:17:09.238415 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238272    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="989f6e36-68cb-455c-89de-cada265e3a3c" containerName="cleanup"
Jul 13 22:17:09.238415 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.238279    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87262d7-e35b-42a4-8595-b1bbfbbd4d68" containerName="main"
Jul 13 22:17:09.240924 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.240906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.244577 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.244550    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d8932b\"/\"default-dockercfg-hdwjt\""
Jul 13 22:17:09.244577 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.244572    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d8932b\"/\"kube-root-ca.crt\""
Jul 13 22:17:09.244785 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.244611    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d8932b\"/\"rate-test-model-d8932b-kserve-self-signed-certs\""
Jul 13 22:17:09.244785 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.244550    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d8932b\"/\"openshift-service-ca.crt\""
Jul 13 22:17:09.248746 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.248718    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"]
Jul 13 22:17:09.261638 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.261607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:17:09.343587 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.343544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-tmp-dir\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.343776 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.343610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-model-cache\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.343776 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.343637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-dshm\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.343776 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.343686    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-home\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.343776 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.343720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29xb\" (UniqueName: \"kubernetes.io/projected/e5f42232-8b6c-4222-8140-8ef5eff756ed-kube-api-access-t29xb\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.343917 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.343807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f42232-8b6c-4222-8140-8ef5eff756ed-tls-certs\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.444865 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.444823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f42232-8b6c-4222-8140-8ef5eff756ed-tls-certs\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.445064 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.444876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-tmp-dir\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.445064 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.444933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-model-cache\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.445064 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.444963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-dshm\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.445064 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.444991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-home\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.445064 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.445034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29xb\" (UniqueName: \"kubernetes.io/projected/e5f42232-8b6c-4222-8140-8ef5eff756ed-kube-api-access-t29xb\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.445416 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.445387    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-tmp-dir\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.445416 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.445410    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-home\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.445537 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.445490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-model-cache\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.447368 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.447349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-dshm\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.447553 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.447537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f42232-8b6c-4222-8140-8ef5eff756ed-tls-certs\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.453573 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.453545    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29xb\" (UniqueName: \"kubernetes.io/projected/e5f42232-8b6c-4222-8140-8ef5eff756ed-kube-api-access-t29xb\") pod \"rate-test-model-d8932b-kserve-55767564dc-jhzzb\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") " pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.563431 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.563334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:09.700550 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:09.700512    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"]
Jul 13 22:17:09.701776 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:17:09.701749    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5f42232_8b6c_4222_8140_8ef5eff756ed.slice/crio-5d4159fa22f4991ad328db8e3a7c0d94d4e46b401421bcfb59a29a6d9a87b14d WatchSource:0}: Error finding container 5d4159fa22f4991ad328db8e3a7c0d94d4e46b401421bcfb59a29a6d9a87b14d: Status 404 returned error can't find the container with id 5d4159fa22f4991ad328db8e3a7c0d94d4e46b401421bcfb59a29a6d9a87b14d
Jul 13 22:17:10.102327 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:10.102285    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb" event={"ID":"e5f42232-8b6c-4222-8140-8ef5eff756ed","Type":"ContainerStarted","Data":"ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309"}
Jul 13 22:17:10.102327 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:10.102332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb" event={"ID":"e5f42232-8b6c-4222-8140-8ef5eff756ed","Type":"ContainerStarted","Data":"5d4159fa22f4991ad328db8e3a7c0d94d4e46b401421bcfb59a29a6d9a87b14d"}
Jul 13 22:17:10.120289 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:10.120234    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb" podStartSLOduration=1.120218894 podStartE2EDuration="1.120218894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:17:10.118474017 +0000 UTC m=+2871.274867856" watchObservedRunningTime="2026-07-13 22:17:10.120218894 +0000 UTC m=+2871.276612735"
Jul 13 22:17:13.549317 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:13.549233    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:17:19.564250 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:19.564213    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:19.564642 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:19.564268    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:19.576909 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:19.576879    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:20.165015 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:20.164981    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:17:28.708174 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:28.708126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:17:28.808765 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:28.808731    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:17:33.847378 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:33.847341    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:17:36.663786 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:36.663745    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:17:36.770265 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:36.770229    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:17:49.049661 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:49.046879    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:17:53.588278 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:53.588229    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:17:54.927438 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:54.927399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:17:55.923419 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:55.923388    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"]
Jul 13 22:17:55.943796 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:17:55.943756    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:02.944305 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:02.944209    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:18:10.946224 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:10.946145    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb" podUID="e5f42232-8b6c-4222-8140-8ef5eff756ed" containerName="main" containerID="cri-o://ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309" gracePeriod=45
Jul 13 22:18:11.200864 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.200794    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:18:11.290587 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.290555    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-model-cache\") pod \"e5f42232-8b6c-4222-8140-8ef5eff756ed\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") "
Jul 13 22:18:11.290772 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.290611    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-home\") pod \"e5f42232-8b6c-4222-8140-8ef5eff756ed\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") "
Jul 13 22:18:11.290772 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.290631    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f42232-8b6c-4222-8140-8ef5eff756ed-tls-certs\") pod \"e5f42232-8b6c-4222-8140-8ef5eff756ed\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") "
Jul 13 22:18:11.290772 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.290681    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-tmp-dir\") pod \"e5f42232-8b6c-4222-8140-8ef5eff756ed\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") "
Jul 13 22:18:11.290772 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.290709    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-dshm\") pod \"e5f42232-8b6c-4222-8140-8ef5eff756ed\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") "
Jul 13 22:18:11.290772 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.290734    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t29xb\" (UniqueName: \"kubernetes.io/projected/e5f42232-8b6c-4222-8140-8ef5eff756ed-kube-api-access-t29xb\") pod \"e5f42232-8b6c-4222-8140-8ef5eff756ed\" (UID: \"e5f42232-8b6c-4222-8140-8ef5eff756ed\") "
Jul 13 22:18:11.291047 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.290848    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5f42232-8b6c-4222-8140-8ef5eff756ed" (UID: "e5f42232-8b6c-4222-8140-8ef5eff756ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:18:11.291047 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.290935    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-home" (OuterVolumeSpecName: "home") pod "e5f42232-8b6c-4222-8140-8ef5eff756ed" (UID: "e5f42232-8b6c-4222-8140-8ef5eff756ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:18:11.291047 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.290980    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5f42232-8b6c-4222-8140-8ef5eff756ed" (UID: "e5f42232-8b6c-4222-8140-8ef5eff756ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:18:11.291047 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.290993    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-model-cache\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:11.292927 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.292904    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-dshm" (OuterVolumeSpecName: "dshm") pod "e5f42232-8b6c-4222-8140-8ef5eff756ed" (UID: "e5f42232-8b6c-4222-8140-8ef5eff756ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 22:18:11.293006 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.292945    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f42232-8b6c-4222-8140-8ef5eff756ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5f42232-8b6c-4222-8140-8ef5eff756ed" (UID: "e5f42232-8b6c-4222-8140-8ef5eff756ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:18:11.293051 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.293019    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f42232-8b6c-4222-8140-8ef5eff756ed-kube-api-access-t29xb" (OuterVolumeSpecName: "kube-api-access-t29xb") pod "e5f42232-8b6c-4222-8140-8ef5eff756ed" (UID: "e5f42232-8b6c-4222-8140-8ef5eff756ed"). InnerVolumeSpecName "kube-api-access-t29xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:18:11.392139 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.392108    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-home\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:11.392139 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.392134    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f42232-8b6c-4222-8140-8ef5eff756ed-tls-certs\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:11.392343 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.392184    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-tmp-dir\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:11.392343 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.392192    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f42232-8b6c-4222-8140-8ef5eff756ed-dshm\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:11.392343 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.392203    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t29xb\" (UniqueName: \"kubernetes.io/projected/e5f42232-8b6c-4222-8140-8ef5eff756ed-kube-api-access-t29xb\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:11.411649 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.411617    2582 generic.go:356] "Generic (PLEG): container finished" podID="e5f42232-8b6c-4222-8140-8ef5eff756ed" containerID="ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309" exitCode=0
Jul 13 22:18:11.411775 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.411658    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"
Jul 13 22:18:11.411775 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.411705    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb" event={"ID":"e5f42232-8b6c-4222-8140-8ef5eff756ed","Type":"ContainerDied","Data":"ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309"}
Jul 13 22:18:11.411775 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.411746    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb" event={"ID":"e5f42232-8b6c-4222-8140-8ef5eff756ed","Type":"ContainerDied","Data":"5d4159fa22f4991ad328db8e3a7c0d94d4e46b401421bcfb59a29a6d9a87b14d"}
Jul 13 22:18:11.411775 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.411763    2582 scope.go:117] "RemoveContainer" containerID="ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309"
Jul 13 22:18:11.420678 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.420662    2582 scope.go:117] "RemoveContainer" containerID="ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309"
Jul 13 22:18:11.420942 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:18:11.420922    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309\": container with ID starting with ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309 not found: ID does not exist" containerID="ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309"
Jul 13 22:18:11.421012 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.420955    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309"} err="failed to get container status \"ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309\": rpc error: code = NotFound desc = could not find container \"ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309\": container with ID starting with ec6d802de4c0170f98fbcab141eda366ec29c2c928aaaf53ace2e2a4b9b50309 not found: ID does not exist"
Jul 13 22:18:11.428355 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.428334    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"]
Jul 13 22:18:11.431890 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:11.431869    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d8932b/rate-test-model-d8932b-kserve-55767564dc-jhzzb"]
Jul 13 22:18:13.363464 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:13.363433    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5f42232-8b6c-4222-8140-8ef5eff756ed" path="/var/lib/kubelet/pods/e5f42232-8b6c-4222-8140-8ef5eff756ed/volumes"
Jul 13 22:18:14.733393 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:14.733351    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn"]
Jul 13 22:18:14.733900 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:14.733876    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f42232-8b6c-4222-8140-8ef5eff756ed" containerName="main"
Jul 13 22:18:14.733982 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:14.733904    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f42232-8b6c-4222-8140-8ef5eff756ed" containerName="main"
Jul 13 22:18:14.734061 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:14.734046    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f42232-8b6c-4222-8140-8ef5eff756ed" containerName="main"
Jul 13 22:18:14.737365 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:14.737340    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn"
Jul 13 22:18:14.739877 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:14.739850    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:18:14.743356 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:14.743325    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn"]
Jul 13 22:18:14.823038 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:14.823007    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/582f2127-bfdc-48f4-9bd5-94c13560ca2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn\" (UID: \"582f2127-bfdc-48f4-9bd5-94c13560ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn"
Jul 13 22:18:14.924052 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:14.924004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/582f2127-bfdc-48f4-9bd5-94c13560ca2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn\" (UID: \"582f2127-bfdc-48f4-9bd5-94c13560ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn"
Jul 13 22:18:14.924704 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:14.924684    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/582f2127-bfdc-48f4-9bd5-94c13560ca2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn\" (UID: \"582f2127-bfdc-48f4-9bd5-94c13560ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn"
Jul 13 22:18:15.050181 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:15.050057    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn"
Jul 13 22:18:15.184887 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:15.184854    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn"]
Jul 13 22:18:15.186443 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:18:15.186414    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582f2127_bfdc_48f4_9bd5_94c13560ca2a.slice/crio-c036e4bb8b12dfd213e2581062cb782dcda1a944c6dbb7783f8b6351b7b31875 WatchSource:0}: Error finding container c036e4bb8b12dfd213e2581062cb782dcda1a944c6dbb7783f8b6351b7b31875: Status 404 returned error can't find the container with id c036e4bb8b12dfd213e2581062cb782dcda1a944c6dbb7783f8b6351b7b31875
Jul 13 22:18:15.242203 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:15.242146    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:15.338394 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:15.338311    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:15.434516 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:15.434466    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn" event={"ID":"582f2127-bfdc-48f4-9bd5-94c13560ca2a","Type":"ContainerStarted","Data":"c036e4bb8b12dfd213e2581062cb782dcda1a944c6dbb7783f8b6351b7b31875"}
Jul 13 22:18:16.440711 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:16.440675    2582 generic.go:356] "Generic (PLEG): container finished" podID="582f2127-bfdc-48f4-9bd5-94c13560ca2a" containerID="9bc5e48a5e60187d414c26efa230704c77d3c4f68729d75e00131147017c1ad0" exitCode=0
Jul 13 22:18:16.440711 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:16.440713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn" event={"ID":"582f2127-bfdc-48f4-9bd5-94c13560ca2a","Type":"ContainerDied","Data":"9bc5e48a5e60187d414c26efa230704c77d3c4f68729d75e00131147017c1ad0"}
Jul 13 22:18:17.583604 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:17.583578    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn"
Jul 13 22:18:17.646511 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:17.646469    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/582f2127-bfdc-48f4-9bd5-94c13560ca2a-maas-api-service-ca\") pod \"582f2127-bfdc-48f4-9bd5-94c13560ca2a\" (UID: \"582f2127-bfdc-48f4-9bd5-94c13560ca2a\") "
Jul 13 22:18:17.646893 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:17.646863    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582f2127-bfdc-48f4-9bd5-94c13560ca2a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "582f2127-bfdc-48f4-9bd5-94c13560ca2a" (UID: "582f2127-bfdc-48f4-9bd5-94c13560ca2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:18:17.747804 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:17.747771    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/582f2127-bfdc-48f4-9bd5-94c13560ca2a-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:18.456123 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:18.456093    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn"
Jul 13 22:18:18.456341 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:18.456125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn" event={"ID":"582f2127-bfdc-48f4-9bd5-94c13560ca2a","Type":"ContainerDied","Data":"c036e4bb8b12dfd213e2581062cb782dcda1a944c6dbb7783f8b6351b7b31875"}
Jul 13 22:18:18.456341 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:18.456188    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c036e4bb8b12dfd213e2581062cb782dcda1a944c6dbb7783f8b6351b7b31875"
Jul 13 22:18:25.111444 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.111408    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"]
Jul 13 22:18:25.111905 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.111853    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk" podUID="2ac79734-8ba9-4de9-adda-be140fc8f3f4" containerName="maas-api" containerID="cri-o://6a4781d2756ca732822c60c8a088915b6907a75eba451c264fec0e632cf4834d" gracePeriod=30
Jul 13 22:18:25.138070 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.138027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:25.138298 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.138034    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7"]
Jul 13 22:18:25.140536 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.140502    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f89343-29733015-lcxj7"]
Jul 13 22:18:25.363342 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.363267    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc81a9be-d077-43ec-8e18-f33cb60ab425" path="/var/lib/kubelet/pods/cc81a9be-d077-43ec-8e18-f33cb60ab425/volumes"
Jul 13 22:18:25.496593 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.496565    2582 generic.go:356] "Generic (PLEG): container finished" podID="2ac79734-8ba9-4de9-adda-be140fc8f3f4" containerID="6a4781d2756ca732822c60c8a088915b6907a75eba451c264fec0e632cf4834d" exitCode=0
Jul 13 22:18:25.496748 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.496622    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk" event={"ID":"2ac79734-8ba9-4de9-adda-be140fc8f3f4","Type":"ContainerDied","Data":"6a4781d2756ca732822c60c8a088915b6907a75eba451c264fec0e632cf4834d"}
Jul 13 22:18:25.562893 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.562868    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:18:25.722345 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.722263    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9fcb\" (UniqueName: \"kubernetes.io/projected/2ac79734-8ba9-4de9-adda-be140fc8f3f4-kube-api-access-x9fcb\") pod \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") "
Jul 13 22:18:25.722496 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.722397    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac79734-8ba9-4de9-adda-be140fc8f3f4-openshift-service-ca\") pod \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") "
Jul 13 22:18:25.722496 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.722432    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac79734-8ba9-4de9-adda-be140fc8f3f4-maas-api-tls\") pod \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\" (UID: \"2ac79734-8ba9-4de9-adda-be140fc8f3f4\") "
Jul 13 22:18:25.722802 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.722766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac79734-8ba9-4de9-adda-be140fc8f3f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ac79734-8ba9-4de9-adda-be140fc8f3f4" (UID: "2ac79734-8ba9-4de9-adda-be140fc8f3f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:18:25.724294 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.724266    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac79734-8ba9-4de9-adda-be140fc8f3f4-kube-api-access-x9fcb" (OuterVolumeSpecName: "kube-api-access-x9fcb") pod "2ac79734-8ba9-4de9-adda-be140fc8f3f4" (UID: "2ac79734-8ba9-4de9-adda-be140fc8f3f4"). InnerVolumeSpecName "kube-api-access-x9fcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:18:25.724399 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.724373    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac79734-8ba9-4de9-adda-be140fc8f3f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ac79734-8ba9-4de9-adda-be140fc8f3f4" (UID: "2ac79734-8ba9-4de9-adda-be140fc8f3f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:18:25.823209 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.823176    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac79734-8ba9-4de9-adda-be140fc8f3f4-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:25.823372 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.823203    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9fcb\" (UniqueName: \"kubernetes.io/projected/2ac79734-8ba9-4de9-adda-be140fc8f3f4-kube-api-access-x9fcb\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:25.823372 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:25.823239    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac79734-8ba9-4de9-adda-be140fc8f3f4-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:26.502733 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:26.502691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk" event={"ID":"2ac79734-8ba9-4de9-adda-be140fc8f3f4","Type":"ContainerDied","Data":"6e26d7ea591a6b3f4b9afba64da7ce88e29522990ed710b31c62427253e3bc1c"}
Jul 13 22:18:26.502733 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:26.502713    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"
Jul 13 22:18:26.503216 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:26.502753    2582 scope.go:117] "RemoveContainer" containerID="6a4781d2756ca732822c60c8a088915b6907a75eba451c264fec0e632cf4834d"
Jul 13 22:18:26.525261 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:26.525230    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"]
Jul 13 22:18:26.528837 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:26.528800    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f89343-6ffcf9554-hcprk"]
Jul 13 22:18:27.363379 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:27.363344    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ac79734-8ba9-4de9-adda-be140fc8f3f4" path="/var/lib/kubelet/pods/2ac79734-8ba9-4de9-adda-be140fc8f3f4/volumes"
Jul 13 22:18:28.551457 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:28.551419    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:18:29.824297 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:29.824267    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:35.667579 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:35.667537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:35.735643 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:35.735611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:40.504885 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.504851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:40.749427 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.749391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:40.870828 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.870743    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv"]
Jul 13 22:18:40.871098 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.871086    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582f2127-bfdc-48f4-9bd5-94c13560ca2a" containerName="revoke-keys"
Jul 13 22:18:40.871143 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.871100    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="582f2127-bfdc-48f4-9bd5-94c13560ca2a" containerName="revoke-keys"
Jul 13 22:18:40.871143 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.871117    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac79734-8ba9-4de9-adda-be140fc8f3f4" containerName="maas-api"
Jul 13 22:18:40.871143 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.871123    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac79734-8ba9-4de9-adda-be140fc8f3f4" containerName="maas-api"
Jul 13 22:18:40.871251 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.871190    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="582f2127-bfdc-48f4-9bd5-94c13560ca2a" containerName="revoke-keys"
Jul 13 22:18:40.871251 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.871201    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac79734-8ba9-4de9-adda-be140fc8f3f4" containerName="maas-api"
Jul 13 22:18:40.874403 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.874387    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv"
Jul 13 22:18:40.876994 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.876971    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fx7c\""
Jul 13 22:18:40.881011 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:40.880987    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv"]
Jul 13 22:18:41.065213 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:41.065171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcfe944c-7384-4e00-95af-097201a85453-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv\" (UID: \"dcfe944c-7384-4e00-95af-097201a85453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv"
Jul 13 22:18:41.166489 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:41.166400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcfe944c-7384-4e00-95af-097201a85453-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv\" (UID: \"dcfe944c-7384-4e00-95af-097201a85453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv"
Jul 13 22:18:41.167031 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:41.167000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcfe944c-7384-4e00-95af-097201a85453-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv\" (UID: \"dcfe944c-7384-4e00-95af-097201a85453\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv"
Jul 13 22:18:41.186361 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:41.186332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv"
Jul 13 22:18:41.324636 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:41.324609    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv"]
Jul 13 22:18:41.327230 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:18:41.327196    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfe944c_7384_4e00_95af_097201a85453.slice/crio-0b5ec1b3ff4d94ce66b6a7cf3e4e25e5cca26f85c82e751b915f05b7da9e1b32 WatchSource:0}: Error finding container 0b5ec1b3ff4d94ce66b6a7cf3e4e25e5cca26f85c82e751b915f05b7da9e1b32: Status 404 returned error can't find the container with id 0b5ec1b3ff4d94ce66b6a7cf3e4e25e5cca26f85c82e751b915f05b7da9e1b32
Jul 13 22:18:41.369765 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:41.369744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:41.469837 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:41.469750    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:41.582091 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:41.582053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv" event={"ID":"dcfe944c-7384-4e00-95af-097201a85453","Type":"ContainerStarted","Data":"0b5ec1b3ff4d94ce66b6a7cf3e4e25e5cca26f85c82e751b915f05b7da9e1b32"}
Jul 13 22:18:42.589267 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:42.589222    2582 generic.go:356] "Generic (PLEG): container finished" podID="dcfe944c-7384-4e00-95af-097201a85453" containerID="e6f3754f3db5c7602badc52a77b416d29493149cf02d4fdaf77c55383e6241fe" exitCode=0
Jul 13 22:18:42.589693 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:42.589299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv" event={"ID":"dcfe944c-7384-4e00-95af-097201a85453","Type":"ContainerDied","Data":"e6f3754f3db5c7602badc52a77b416d29493149cf02d4fdaf77c55383e6241fe"}
Jul 13 22:18:43.722573 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:43.722548    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv"
Jul 13 22:18:43.792744 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:43.792710    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcfe944c-7384-4e00-95af-097201a85453-maas-api-service-ca\") pod \"dcfe944c-7384-4e00-95af-097201a85453\" (UID: \"dcfe944c-7384-4e00-95af-097201a85453\") "
Jul 13 22:18:43.793065 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:43.793042    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfe944c-7384-4e00-95af-097201a85453-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcfe944c-7384-4e00-95af-097201a85453" (UID: "dcfe944c-7384-4e00-95af-097201a85453"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:18:43.893707 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:43.893630    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcfe944c-7384-4e00-95af-097201a85453-maas-api-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:44.601618 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:44.601585    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv"
Jul 13 22:18:44.601797 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:44.601623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv" event={"ID":"dcfe944c-7384-4e00-95af-097201a85453","Type":"ContainerDied","Data":"0b5ec1b3ff4d94ce66b6a7cf3e4e25e5cca26f85c82e751b915f05b7da9e1b32"}
Jul 13 22:18:44.601797 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:44.601676    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b5ec1b3ff4d94ce66b6a7cf3e4e25e5cca26f85c82e751b915f05b7da9e1b32"
Jul 13 22:18:51.261450 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.261406    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"]
Jul 13 22:18:51.261981 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.261703    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd" podUID="2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7" containerName="maas-api" containerID="cri-o://3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf" gracePeriod=30
Jul 13 22:18:51.288332 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.288302    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql"]
Jul 13 22:18:51.291127 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.291096    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d8932b-29733015-wdhql"]
Jul 13 22:18:51.294470 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.294444    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:18:51.362982 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.362940    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bf5c175-46c5-4032-a341-09396406df71" path="/var/lib/kubelet/pods/2bf5c175-46c5-4032-a341-09396406df71/volumes"
Jul 13 22:18:51.516320 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.516254    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:18:51.559048 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.558987    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l72ql\" (UniqueName: \"kubernetes.io/projected/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-kube-api-access-l72ql\") pod \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") "
Jul 13 22:18:51.559293 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.559112    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-openshift-service-ca\") pod \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") "
Jul 13 22:18:51.559293 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.559185    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-maas-api-tls\") pod \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\" (UID: \"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7\") "
Jul 13 22:18:51.559546 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.559514    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7" (UID: "2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 22:18:51.561311 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.561273    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7" (UID: "2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 22:18:51.561311 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.561279    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-kube-api-access-l72ql" (OuterVolumeSpecName: "kube-api-access-l72ql") pod "2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7" (UID: "2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7"). InnerVolumeSpecName "kube-api-access-l72ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 22:18:51.639441 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.639405    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7" containerID="3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf" exitCode=0
Jul 13 22:18:51.639630 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.639454    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"
Jul 13 22:18:51.639630 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.639498    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd" event={"ID":"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7","Type":"ContainerDied","Data":"3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf"}
Jul 13 22:18:51.639630 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.639556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd" event={"ID":"2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7","Type":"ContainerDied","Data":"0e6221535d406d0b073b592608415d2bd5dedfeaac6f62dde63ca71adcc0a591"}
Jul 13 22:18:51.639630 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.639589    2582 scope.go:117] "RemoveContainer" containerID="3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf"
Jul 13 22:18:51.649955 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.649933    2582 scope.go:117] "RemoveContainer" containerID="3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf"
Jul 13 22:18:51.650302 ip-10-0-140-193 kubenswrapper[2582]: E0713 22:18:51.650275    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf\": container with ID starting with 3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf not found: ID does not exist" containerID="3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf"
Jul 13 22:18:51.650403 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.650311    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf"} err="failed to get container status \"3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf\": rpc error: code = NotFound desc = could not find container \"3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf\": container with ID starting with 3316ef1af4057d79259c0bc5c94a9187e3520f2f8f1c98b64624ffce23566bbf not found: ID does not exist"
Jul 13 22:18:51.660807 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.660768    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l72ql\" (UniqueName: \"kubernetes.io/projected/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-kube-api-access-l72ql\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:51.660807 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.660805    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-openshift-service-ca\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:51.660807 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.660817    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7-maas-api-tls\") on node \"ip-10-0-140-193.ec2.internal\" DevicePath \"\""
Jul 13 22:18:51.661989 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.661969    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"]
Jul 13 22:18:51.667582 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:51.667558    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d8932b-845bcdb7d6-wjrmd"]
Jul 13 22:18:53.363499 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:53.363466    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7" path="/var/lib/kubelet/pods/2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7/volumes"
Jul 13 22:18:54.548139 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:54.548101    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-np646"]
Jul 13 22:18:55.945837 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:18:55.945805    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:19:01.969822 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:01.969782    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:19:02.030567 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:02.030521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:19:06.643443 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:06.643408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:19:06.883395 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:06.883363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:19:12.461911 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:12.461881    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79f5876b6c-dl5fc_e49c6573-43e7-4736-b00e-c4cc48492b59/manager/0.log"
Jul 13 22:19:13.063662 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:13.063635    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5db8865497-l5p4v_8b9d335d-d26b-4539-b5b0-4e7e14d4e99e/manager/0.log"
Jul 13 22:19:13.069645 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:13.069624    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5db8865497-l5p4v_8b9d335d-d26b-4539-b5b0-4e7e14d4e99e/copy-manifests/0.log"
Jul 13 22:19:13.559350 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:13.559316    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732985-j4z9w_767c4252-12fa-41a6-a81a-e73d38ab0162/cleanup/0.log"
Jul 13 22:19:13.673740 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:13.673714    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733000-kxznn_b33f5af5-6a1f-4cd0-8d24-ca295a4b0f6c/cleanup/0.log"
Jul 13 22:19:13.786635 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:13.786604    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733015-6jqd2_989f6e36-68cb-455c-89de-cada265e3a3c/cleanup/0.log"
Jul 13 22:19:13.899547 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:13.899455    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-30bfd308-d66pv_6a337783-6294-4eb9-a439-886c7f7cd349/revoke-keys/0.log"
Jul 13 22:19:14.013597 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:14.013573    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-56bf06f8-78bqc_fea04fa6-3653-4578-b357-9260281d910a/revoke-keys/0.log"
Jul 13 22:19:14.130410 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:14.130382    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-785879-l88xb_912ba5d2-ec45-4a89-a8d3-7159bf7f59e8/revoke-keys/0.log"
Jul 13 22:19:14.270339 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:14.270312    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-9dff5a-5klwh_6083269a-308c-4dfc-96c0-d28a9639a7d0/revoke-keys/0.log"
Jul 13 22:19:14.393047 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:14.393012    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-628bb6aa-8zft8_c9358e6a-f5a5-4ae5-9f45-40f6014b6ee1/revoke-keys/0.log"
Jul 13 22:19:14.508892 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:14.508865    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-160510-8flls_beeb0b58-2f85-4aac-91d1-08aca9491e58/revoke-keys/0.log"
Jul 13 22:19:14.628703 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:14.628627    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-3394b0-9tcsj_b2cb5891-2e79-40f8-b956-dde8f7e3d4c3/revoke-keys/0.log"
Jul 13 22:19:14.721247 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:14.721197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:19:14.751005 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:14.750981    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-7e874d59-569xh_71c8a58d-9860-474e-9b74-25f77113d5fd/revoke-keys/0.log"
Jul 13 22:19:14.865635 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:14.865605    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-06fcd095-vzggx_0023c91c-ea2f-4f3d-ae34-d31c4a8a842b/revoke-keys/0.log"
Jul 13 22:19:14.980927 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:14.980903    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-a941a48d-7msx4_2607e304-03ae-480f-bfe0-57a15a8c204b/revoke-keys/0.log"
Jul 13 22:19:15.098217 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:15.098188    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-f89343-m6xbn_582f2127-bfdc-48f4-9bd5-94c13560ca2a/revoke-keys/0.log"
Jul 13 22:19:15.213178 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:15.213122    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-d8932b-zkfrv_dcfe944c-7384-4e00-95af-097201a85453/revoke-keys/0.log"
Jul 13 22:19:16.877978 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:16.877945    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xcrkn_87ea54b2-4db8-4e84-9118-af4827ed9353/manager/0.log"
Jul 13 22:19:17.351492 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:17.351466    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-np646_5a1b0068-ddc3-4234-83cc-151c97892bdc/limitador/0.log"
Jul 13 22:19:17.950341 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:17.950307    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zvmxq_e214029a-2b87-4505-a38d-90a5fd7bfae7/discovery/0.log"
Jul 13 22:19:18.668601 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:18.668568    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56948c7b44-sgwgk_6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6/router/0.log"
Jul 13 22:19:19.137563 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:19.137531    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wtjzj_476e1bc9-2693-44b7-a940-40db993c4c91/main/0.log"
Jul 13 22:19:19.451813 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:19.451731    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 22:19:19.456464 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:19.456440    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 22:19:19.459111 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:19.459091    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 22:19:19.463497 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:19.463479    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm77k_0fb821cc-e52a-468e-b50b-afc5e461d6c5/ovn-acl-logging/0.log"
Jul 13 22:19:23.581911 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:23.581875    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:19:23.682232 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:23.682197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log" limit=1024
Jul 13 22:19:26.473492 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:26.473459    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x6tw2_4271181f-f9a6-4bc2-ab11-9e21b0afebd2/global-pull-secret-syncer/0.log"
Jul 13 22:19:26.521444 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:26.521416    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pmd7k_9dd52d08-2a36-4291-ad30-52d7bb0ba254/konnectivity-agent/0.log"
Jul 13 22:19:26.642354 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:26.642326    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-193.ec2.internal_f7766334a1c1b5307e0876e60a26fe5a/haproxy/0.log"
Jul 13 22:19:31.427005 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:31.426974    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xcrkn_87ea54b2-4db8-4e84-9118-af4827ed9353/manager/0.log"
Jul 13 22:19:31.712475 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:31.712397    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-np646_5a1b0068-ddc3-4234-83cc-151c97892bdc/limitador/0.log"
Jul 13 22:19:33.590209 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:33.590116    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78b84b988f-t7xp6_0e3bf464-3a62-41b3-9f4d-ced0b4f179ef/metrics-server/0.log"
Jul 13 22:19:33.812811 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:33.812779    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xx6xg_c2fd3764-1986-4104-98ec-58a05d107526/node-exporter/0.log"
Jul 13 22:19:33.836271 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:33.836219    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xx6xg_c2fd3764-1986-4104-98ec-58a05d107526/kube-rbac-proxy/0.log"
Jul 13 22:19:33.860139 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:33.860067    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xx6xg_c2fd3764-1986-4104-98ec-58a05d107526/init-textfile/0.log"
Jul 13 22:19:33.979015 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:33.978983    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0c89e99-d717-4011-b22a-ba53cba3597c/prometheus/0.log"
Jul 13 22:19:34.001052 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.001030    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0c89e99-d717-4011-b22a-ba53cba3597c/config-reloader/0.log"
Jul 13 22:19:34.024480 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.024457    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0c89e99-d717-4011-b22a-ba53cba3597c/thanos-sidecar/0.log"
Jul 13 22:19:34.045985 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.045961    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0c89e99-d717-4011-b22a-ba53cba3597c/kube-rbac-proxy-web/0.log"
Jul 13 22:19:34.068282 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.068262    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0c89e99-d717-4011-b22a-ba53cba3597c/kube-rbac-proxy/0.log"
Jul 13 22:19:34.092639 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.092620    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0c89e99-d717-4011-b22a-ba53cba3597c/kube-rbac-proxy-thanos/0.log"
Jul 13 22:19:34.116008 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.115950    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0c89e99-d717-4011-b22a-ba53cba3597c/init-config-reloader/0.log"
Jul 13 22:19:34.305682 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.305653    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6997d46fcf-gjb5g_a4b29830-01ac-438b-86fe-0291e9cd1464/thanos-query/0.log"
Jul 13 22:19:34.326957 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.326930    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6997d46fcf-gjb5g_a4b29830-01ac-438b-86fe-0291e9cd1464/kube-rbac-proxy-web/0.log"
Jul 13 22:19:34.347027 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.347002    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6997d46fcf-gjb5g_a4b29830-01ac-438b-86fe-0291e9cd1464/kube-rbac-proxy/0.log"
Jul 13 22:19:34.378492 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.378430    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6997d46fcf-gjb5g_a4b29830-01ac-438b-86fe-0291e9cd1464/prom-label-proxy/0.log"
Jul 13 22:19:34.399891 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.399866    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6997d46fcf-gjb5g_a4b29830-01ac-438b-86fe-0291e9cd1464/kube-rbac-proxy-rules/0.log"
Jul 13 22:19:34.424946 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:34.424924    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6997d46fcf-gjb5g_a4b29830-01ac-438b-86fe-0291e9cd1464/kube-rbac-proxy-metrics/0.log"
Jul 13 22:19:36.108688 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.108642    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"]
Jul 13 22:19:36.109138 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.109102    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcfe944c-7384-4e00-95af-097201a85453" containerName="revoke-keys"
Jul 13 22:19:36.109138 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.109124    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcfe944c-7384-4e00-95af-097201a85453" containerName="revoke-keys"
Jul 13 22:19:36.109469 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.109144    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7" containerName="maas-api"
Jul 13 22:19:36.109469 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.109168    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7" containerName="maas-api"
Jul 13 22:19:36.109469 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.109458    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c6b0fb6-7897-4ef2-ab20-c72d96b8d1a7" containerName="maas-api"
Jul 13 22:19:36.109568 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.109485    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcfe944c-7384-4e00-95af-097201a85453" containerName="revoke-keys"
Jul 13 22:19:36.112728 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.112705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.115426 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.115397    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjrg5\"/\"openshift-service-ca.crt\""
Jul 13 22:19:36.115979 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.115869    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjrg5\"/\"kube-root-ca.crt\""
Jul 13 22:19:36.116691 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.116671    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wjrg5\"/\"default-dockercfg-wn26n\""
Jul 13 22:19:36.118483 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.118459    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"]
Jul 13 22:19:36.154575 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.154539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-sys\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.154751 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.154580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr7f7\" (UniqueName: \"kubernetes.io/projected/9c85fc80-00ca-412c-9efe-2c7692b5af89-kube-api-access-mr7f7\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.154751 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.154609    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-proc\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.154751 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.154636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-podres\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.154904 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.154767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-lib-modules\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.236193 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.236142    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/2.log"
Jul 13 22:19:36.240522 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.240498    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nslf5_46df3134-e1d1-42e7-aacc-ad226f9d84a6/console-operator/3.log"
Jul 13 22:19:36.255389 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.255363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-lib-modules\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.255490 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.255435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-sys\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.255490 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.255465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr7f7\" (UniqueName: \"kubernetes.io/projected/9c85fc80-00ca-412c-9efe-2c7692b5af89-kube-api-access-mr7f7\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.255606 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.255504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-proc\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.255606 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.255538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-sys\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.255606 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.255546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-podres\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.255606 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.255575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-proc\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.255606 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.255540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-lib-modules\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.255789 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.255645    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c85fc80-00ca-412c-9efe-2c7692b5af89-podres\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.263266 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.263248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr7f7\" (UniqueName: \"kubernetes.io/projected/9c85fc80-00ca-412c-9efe-2c7692b5af89-kube-api-access-mr7f7\") pod \"perf-node-gather-daemonset-frdts\" (UID: \"9c85fc80-00ca-412c-9efe-2c7692b5af89\") " pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.424394 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.424306    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.550639 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.550607    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"]
Jul 13 22:19:36.551783 ip-10-0-140-193 kubenswrapper[2582]: W0713 22:19:36.551743    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9c85fc80_00ca_412c_9efe_2c7692b5af89.slice/crio-0f18783ec51121aa7c2f1747e9440b86cb5914d52fcb98f5abe3243960ca8b09 WatchSource:0}: Error finding container 0f18783ec51121aa7c2f1747e9440b86cb5914d52fcb98f5abe3243960ca8b09: Status 404 returned error can't find the container with id 0f18783ec51121aa7c2f1747e9440b86cb5914d52fcb98f5abe3243960ca8b09
Jul 13 22:19:36.882170 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.882125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts" event={"ID":"9c85fc80-00ca-412c-9efe-2c7692b5af89","Type":"ContainerStarted","Data":"0415ae3c24a783709d2c176d845cfea4f49ae0ead8c4e12314d72d8493d6c052"}
Jul 13 22:19:36.882398 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.882199    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:36.882398 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.882219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts" event={"ID":"9c85fc80-00ca-412c-9efe-2c7692b5af89","Type":"ContainerStarted","Data":"0f18783ec51121aa7c2f1747e9440b86cb5914d52fcb98f5abe3243960ca8b09"}
Jul 13 22:19:36.898756 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:36.898704    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts" podStartSLOduration=0.898689585 podStartE2EDuration="898.689585ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:19:36.896614332 +0000 UTC m=+3018.053008174" watchObservedRunningTime="2026-07-13 22:19:36.898689585 +0000 UTC m=+3018.055083425"
Jul 13 22:19:38.376960 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:38.376933    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tvf8f_7e46539f-e0bc-4eea-a0ca-e65633ccf553/dns/0.log"
Jul 13 22:19:38.398096 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:38.398064    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tvf8f_7e46539f-e0bc-4eea-a0ca-e65633ccf553/kube-rbac-proxy/0.log"
Jul 13 22:19:38.497047 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:38.497017    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rh74n_4473539f-ed4b-437c-9814-8ee29ac74bf6/dns-node-resolver/0.log"
Jul 13 22:19:39.032772 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:39.032739    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c7b7959d5-wvnrk_9d23ddb8-f815-405d-beb7-e0e83108a305/registry/0.log"
Jul 13 22:19:39.100784 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:39.100757    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tbqlb_721b2fc5-6fe6-49a2-abaf-ea836e1d93bb/node-ca/0.log"
Jul 13 22:19:40.144377 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:40.144343    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zvmxq_e214029a-2b87-4505-a38d-90a5fd7bfae7/discovery/0.log"
Jul 13 22:19:40.363193 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:40.363141    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56948c7b44-sgwgk_6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6/router/0.log"
Jul 13 22:19:41.046793 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:41.046762    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9znbk_f3d9f579-5163-4bf3-906e-e497ca6d4d89/serve-healthcheck-canary/0.log"
Jul 13 22:19:41.531211 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:41.531182    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-hx5cc_5fdf28c8-beba-416a-9165-66a689308580/insights-operator/0.log"
Jul 13 22:19:41.531651 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:41.531573    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-hx5cc_5fdf28c8-beba-416a-9165-66a689308580/insights-operator/1.log"
Jul 13 22:19:41.554409 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:41.554385    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-79grf_7f5a438a-b79a-4fe8-9621-6ec09a318089/kube-rbac-proxy/0.log"
Jul 13 22:19:41.575238 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:41.575214    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-79grf_7f5a438a-b79a-4fe8-9621-6ec09a318089/exporter/0.log"
Jul 13 22:19:41.595875 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:41.595856    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-79grf_7f5a438a-b79a-4fe8-9621-6ec09a318089/extractor/0.log"
Jul 13 22:19:42.896834 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:42.896798    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wjrg5/perf-node-gather-daemonset-frdts"
Jul 13 22:19:44.166399 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:44.166364    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79f5876b6c-dl5fc_e49c6573-43e7-4736-b00e-c4cc48492b59/manager/0.log"
Jul 13 22:19:44.483265 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:44.483244    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5db8865497-l5p4v_8b9d335d-d26b-4539-b5b0-4e7e14d4e99e/manager/0.log"
Jul 13 22:19:44.508333 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:44.508306    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5db8865497-l5p4v_8b9d335d-d26b-4539-b5b0-4e7e14d4e99e/copy-manifests/0.log"
Jul 13 22:19:45.800331 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:45.800299    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-789595b4db-zg56j_06f75308-f438-4a9c-9ab9-a2d050827295/manager/0.log"
Jul 13 22:19:46.420619 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:46.420586    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zvmxq_e214029a-2b87-4505-a38d-90a5fd7bfae7/discovery/0.log"
Jul 13 22:19:46.591611 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:46.591584    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56948c7b44-sgwgk_6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6/router/0.log"
Jul 13 22:19:47.781927 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:47.781891    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zvmxq_e214029a-2b87-4505-a38d-90a5fd7bfae7/discovery/0.log"
Jul 13 22:19:47.953582 ip-10-0-140-193 kubenswrapper[2582]: I0713 22:19:47.953555    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56948c7b44-sgwgk_6c8a2d79-ce7e-4eb4-b623-bbe7428c36d6/router/0.log"