Aug 04 19:15:54.988740 ip-10-0-136-99 systemd[1]: Starting Kubernetes Kubelet... Aug 04 19:15:55.348571 ip-10-0-136-99 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 19:15:55.348571 ip-10-0-136-99 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 19:15:55.348571 ip-10-0-136-99 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 19:15:55.348571 ip-10-0-136-99 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 19:15:55.348571 ip-10-0-136-99 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 19:15:55.349434 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.349301 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 19:15:55.356267 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356251 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:55.356267 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356267 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356270 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356274 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356277 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356280 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356283 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356286 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356291 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356294 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356296 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356299 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356303 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356307 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356310 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356314 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356318 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356321 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356325 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356328 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:55.356334 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356331 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356334 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356337 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356340 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356344 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356347 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356350 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356353 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356356 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356359 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356362 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356364 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356367 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356370 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356372 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356375 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356377 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356380 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356383 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356385 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:55.356807 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356388 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356391 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356393 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356396 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356398 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356401 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356404 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356406 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356409 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356411 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356414 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356416 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356419 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356421 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356424 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356427 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356430 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356433 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356435 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356438 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:55.357283 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356441 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356443 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356447 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356449 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356452 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356455 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356457 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356473 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356476 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356479 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356481 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356484 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356488 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356491 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356493 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356496 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356499 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356501 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356504 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356506 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:55.357810 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356509 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356512 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356515 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356517 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356520 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356522 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356526 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356529 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356532 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356534 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356537 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356540 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356542 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356545 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356547 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356550 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356553 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356556 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.356561 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:55.358290 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357249 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357256 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357259 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357262 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357265 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357268 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357271 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357274 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357276 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357279 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357282 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357284 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357287 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357290 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357293 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357296 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357299 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357302 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357306 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:55.358814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357310 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357313 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357316 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357319 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357322 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357324 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357327 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357329 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357332 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357335 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357338 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357340 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357344 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357348 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357351 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357353 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357356 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357359 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357362 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:55.359279 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357364 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357367 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357369 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357372 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357374 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357378 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357383 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357385 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357388 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357391 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357394 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357396 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357399 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357402 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357405 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357408 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357410 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357413 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357415 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:55.359775 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357418 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357420 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357423 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357426 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357428 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357431 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357433 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357435 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357438 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357441 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357443 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357446 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357448 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357451 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357454 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357456 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357482 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357487 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357491 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357496 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:55.360232 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357498 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357501 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357503 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357506 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357515 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357518 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357521 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357524 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357527 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357529 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357532 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357535 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357538 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357540 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357543 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357546 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357548 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357551 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357553 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357556 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357558 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:55.360728 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357561 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357959 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357964 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357968 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357971 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357974 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357977 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357979 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357983 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357986 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357989 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357992 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357994 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.357997 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358000 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358004 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358006 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358012 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358015 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358017 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:55.361229 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358021 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358023 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358026 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358028 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358031 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358034 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358036 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358039 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358041 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358044 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358046 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358049 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358051 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358054 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358056 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358059 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358062 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358064 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358067 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358070 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:55.361726 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358072 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358075 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358077 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358080 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358082 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358085 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358087 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358090 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358093 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358097 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358102 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358105 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358108 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358111 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358114 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358116 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358119 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358122 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358125 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:55.362200 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358128 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358131 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358133 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358136 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358139 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358141 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358146 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358150 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358153 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358155 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358158 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358160 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358163 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358165 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358168 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358170 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358173 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358175 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358178 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358180 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:55.362663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358182 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358185 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358191 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358194 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358197 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358199 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358202 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358204 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358207 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358209 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358212 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358215 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358217 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358220 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358223 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358226 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358228 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358231 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358233 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:55.363144 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358236 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358240 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358308 2580 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358314 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358320 2580 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358325 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358329 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358332 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358336 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358340 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358343 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358346 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358349 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358352 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358356 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358361 2580 flags.go:64] FLAG: --cgroup-root="" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358365 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358368 2580 flags.go:64] FLAG: --client-ca-file="" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358371 2580 flags.go:64] FLAG: --cloud-provider="external" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358373 2580 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358378 2580 flags.go:64] FLAG: --cluster-domain="" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358380 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358383 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 19:15:55.363619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358386 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358389 2580 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358393 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358396 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358399 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358402 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358405 2580 flags.go:64] FLAG: --contention-profiling="false" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358408 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358411 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358414 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358417 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358421 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358423 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358426 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358429 2580 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358432 2580 flags.go:64] FLAG: --enable-server="true" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358435 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358440 2580 flags.go:64] FLAG: --event-burst="100" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358443 2580 flags.go:64] FLAG: --event-qps="50" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358446 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358449 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358451 2580 flags.go:64] FLAG: --eviction-hard="" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358455 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358458 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358478 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 19:15:55.364167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358482 2580 flags.go:64] FLAG: --eviction-soft="" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358485 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358488 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358491 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358494 2580 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358497 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358500 2580 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358503 2580 flags.go:64] FLAG: --feature-gates="" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358506 2580 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358509 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358512 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358515 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358518 2580 flags.go:64] FLAG: --healthz-port="10248" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358521 2580 flags.go:64] FLAG: --help="false" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358524 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358527 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358530 2580 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358533 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358536 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358539 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358542 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358545 2580 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358548 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 19:15:55.364804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358550 2580 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358553 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358556 2580 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358559 2580 flags.go:64] FLAG: --kube-reserved="" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358562 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358565 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358568 2580 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358571 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358575 2580 flags.go:64] FLAG: --lock-file="" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358578 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358581 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358584 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358589 2580 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358592 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358595 2580 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358598 2580 flags.go:64] FLAG: --logging-format="text" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358601 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358604 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358607 2580 flags.go:64] FLAG: --manifest-url="" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358610 2580 flags.go:64] FLAG: --manifest-url-header="" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358614 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358617 2580 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358621 2580 flags.go:64] FLAG: --max-pods="110" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358624 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358627 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 19:15:55.365411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358629 2580 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358632 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358635 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358638 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358641 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358648 2580 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358651 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358654 2580 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358657 2580 flags.go:64] FLAG: --pod-cidr="" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358659 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358665 2580 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358668 2580 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358671 2580 flags.go:64] FLAG: --pods-per-core="0" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358673 2580 flags.go:64] FLAG: --port="10250" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358676 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358679 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a9a2fee9f918fc8a" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358684 2580 flags.go:64] FLAG: --qos-reserved="" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358687 2580 flags.go:64] FLAG: --read-only-port="10255" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358689 2580 flags.go:64] FLAG: --register-node="true" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358693 2580 flags.go:64] FLAG: --register-with-taints="" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358696 2580 flags.go:64] FLAG: --registry-burst="10" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358699 2580 flags.go:64] FLAG: --registry-qps="5" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358702 2580 flags.go:64] FLAG: --reserved-cpus="" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358705 2580 flags.go:64] FLAG: --reserved-memory="" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358708 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 19:15:55.366092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358711 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358714 2580 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358717 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358719 2580 flags.go:64] FLAG: --runonce="false" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358722 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358725 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358728 2580 flags.go:64] FLAG: --seccomp-default="false" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358731 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358734 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358737 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358740 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358743 2580 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358746 2580 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358748 2580 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358751 2580 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358754 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358757 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358759 2580 flags.go:64] FLAG: --system-cgroups="" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358762 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358767 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358770 2580 flags.go:64] FLAG: --tls-cert-file="" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358772 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358778 2580 flags.go:64] FLAG: --tls-min-version="" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358781 2580 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358784 2580 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 19:15:55.367036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358787 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358790 2580 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358793 2580 flags.go:64] FLAG: --v="2" Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358797 2580 flags.go:64] FLAG: --version="false" Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358801 2580 flags.go:64] FLAG: --vmodule="" Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358805 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.358808 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358923 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358927 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358930 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358934 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358938 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358941 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358944 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358947 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358950 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358953 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358955 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358958 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358961 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358964 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:55.367663 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358966 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358969 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358971 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358974 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358977 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358979 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358982 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358985 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358989 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358992 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358995 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.358998 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359000 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359003 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359007 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359009 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359012 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359015 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359018 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359020 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:55.368168 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359023 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359025 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359028 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359031 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359033 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359036 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359038 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359041 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359043 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359046 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359049 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359051 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359054 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359057 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359059 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359061 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359065 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359084 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359088 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:55.368785 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359091 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359095 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359099 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359103 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359105 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359108 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359111 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359114 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359118 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359122 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359125 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359127 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359130 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359133 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359135 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359138 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359140 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359143 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359145 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:55.369233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359148 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359150 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359153 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359155 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359158 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359161 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359163 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359166 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359169 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359171 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359174 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359176 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359179 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359181 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359185 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359188 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359191 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359194 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359196 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359199 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:55.369710 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359202 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359205 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359208 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359211 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359213 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359216 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.359219 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.359726 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.368623 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.368638 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.368657 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.368663 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368726 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368731 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368734 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:55.370191 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368738 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368741 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368744 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368747 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368750 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368752 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368755 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368758 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368760 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368763 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368765 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368768 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368770 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368773 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368775 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368778 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368782 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368784 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368787 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368790 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:55.370591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368793 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368795 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368798 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368801 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368803 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368806 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368809 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368812 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368816 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368820 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368823 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368826 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368829 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368832 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368834 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368837 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368839 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368842 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368845 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368847 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:55.371073 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368850 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368853 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368855 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368859 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368861 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368864 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368867 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368869 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368872 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368874 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368877 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368879 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368882 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368884 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368888 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368891 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368894 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368898 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368901 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:55.371568 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368904 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368907 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368909 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368912 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368914 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368917 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368920 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368922 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368925 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368927 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368930 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368932 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368935 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368937 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368940 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368943 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368945 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368948 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368951 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368953 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:55.372021 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368956 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368958 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368961 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368963 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368967 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368971 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368974 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368976 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368980 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368983 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368986 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368988 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368991 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368994 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368997 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.368999 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:55.372645 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369002 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.369007 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369145 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369150 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369153 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369156 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369159 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369162 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369165 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369168 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369171 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369173 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369176 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369178 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:55.373036 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369181 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369185 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369189 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369192 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369195 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369197 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369200 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369203 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369205 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369208 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369210 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369213 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369216 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369219 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369221 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369224 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369226 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369229 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369232 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369234 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:55.373379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369237 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369240 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369242 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369245 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369248 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369250 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369253 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369255 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369258 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369260 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369263 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369267 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369270 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369273 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369276 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369279 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369281 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369284 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369286 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:55.373862 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369289 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369291 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369293 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369296 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369299 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369302 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369304 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369307 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369309 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369312 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369314 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369317 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369320 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369322 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369325 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369328 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369330 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369333 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369335 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369339 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:55.374311 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369343 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369346 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369348 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369351 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369354 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369356 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369358 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369362 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369364 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369366 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369369 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369372 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369374 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369377 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369379 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369382 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369384 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369387 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369389 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369392 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:55.374814 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369394 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369397 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369399 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369402 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369404 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369407 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369409 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:55.369412 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.369416 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.369533 2580 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.371582 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.373092 2580 server.go:1019] "Starting client certificate rotation" Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.373194 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 19:15:55.375288 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.373235 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 19:15:55.375623 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.375500 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 19:15:55.377318 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.377298 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 19:15:55.389043 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.389026 2580 log.go:25] "Validated CRI v1 runtime API" Aug 04 19:15:55.391384 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.391367 2580 log.go:25] "Validated CRI v1 image API" Aug 04 19:15:55.392631 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.392616 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 19:15:55.395641 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.395610 2580 fs.go:135] Filesystem UUIDs: map[1dbc14a5-c4c4-4bfb-9a54-394a9c9c0cc9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 896c6730-ef02-4a2f-9a32-fe18edd119eb:/dev/nvme0n1p4] Aug 04 19:15:55.395707 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.395639 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 19:15:55.401159 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.401042 2580 manager.go:216] Machine: {Timestamp:2026-08-04 19:15:55.399876794 +0000 UTC m=+0.305477325 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102525 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24afced210a40924021ea6c20dbf1f SystemUUID:ec24afce-d210-a409-2402-1ea6c20dbf1f BootID:b3718cdb-bb1e-455f-b27c-207bbb738c49 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0b:ac:39:47:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0b:ac:39:47:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:24:88:00:e1:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 19:15:55.401159 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.401155 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 19:15:55.401270 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.401226 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 19:15:55.401630 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.401612 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 19:15:55.402050 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.402018 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 19:15:55.402182 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.402054 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 19:15:55.403376 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.403366 2580 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 19:15:55.403415 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.403379 2580 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 19:15:55.403415 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.403386 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 19:15:55.404087 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.404074 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 19:15:55.405131 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.405121 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 19:15:55.408937 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.408903 2580 state_mem.go:36] "Initialized new in-memory state store" Aug 04 19:15:55.409415 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.409398 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 19:15:55.409527 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.409436 2580 kubelet.go:486] "Attempting to sync node with API server" Aug 04 19:15:55.409527 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.409449 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 19:15:55.409527 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.409487 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 19:15:55.409527 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.409501 2580 kubelet.go:398] "Adding apiserver pod source" Aug 04 19:15:55.409527 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.409515 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 19:15:55.410271 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.409934 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 19:15:55.410271 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.409960 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 19:15:55.412321 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.412299 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 19:15:55.414095 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.414081 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 19:15:55.414147 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.414111 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 19:15:55.415402 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415391 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 19:15:55.415450 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415418 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 19:15:55.415450 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415428 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 19:15:55.415450 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415437 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 19:15:55.415450 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415449 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 19:15:55.415583 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415457 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 19:15:55.415583 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415483 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 19:15:55.415583 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415488 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 19:15:55.415583 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415495 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 19:15:55.415583 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415501 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 19:15:55.415583 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415509 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 19:15:55.415583 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.415514 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 19:15:55.416129 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.416119 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 19:15:55.416129 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.416128 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 19:15:55.418966 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.418944 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 19:15:55.419042 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.419025 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 19:15:55.419111 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.419096 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 19:15:55.419479 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.419445 2580 server.go:1268] "Started kubelet" Aug 04 19:15:55.419620 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.419550 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 19:15:55.419746 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.419703 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 19:15:55.419813 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.419767 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 19:15:55.420237 ip-10-0-136-99 systemd[1]: Started Kubernetes Kubelet. Aug 04 19:15:55.421143 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.421117 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 19:15:55.421877 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.421863 2580 server.go:310] "Adding debug handlers to kubelet server" Aug 04 19:15:55.426092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.426071 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bl6sl" Aug 04 19:15:55.426391 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.426363 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 19:15:55.426961 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.426948 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 19:15:55.427014 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.426958 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 19:15:55.427455 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.426238 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-99.ec2.internal.18c8b02c08763c6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-99.ec2.internal,UID:ip-10-0-136-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-99.ec2.internal,},FirstTimestamp:2026-08-04 19:15:55.419417708 +0000 UTC m=+0.325018239,LastTimestamp:2026-08-04 19:15:55.419417708 +0000 UTC m=+0.325018239,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-99.ec2.internal,}" Aug 04 19:15:55.427455 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.427136 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:55.427632 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.427616 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 19:15:55.427683 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.427619 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 19:15:55.427683 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.427646 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 19:15:55.427790 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.427778 2580 reconstruct.go:97] "Volume reconstruction finished" Aug 04 19:15:55.427845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.427793 2580 reconciler.go:29] "Reconciler: start to sync state" Aug 04 19:15:55.427845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.427831 2580 factory.go:55] Registering systemd factory Aug 04 19:15:55.427972 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.427868 2580 factory.go:223] Registration of the systemd container factory successfully Aug 04 19:15:55.428114 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.428095 2580 factory.go:153] Registering CRI-O factory Aug 04 19:15:55.428114 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.428115 2580 factory.go:223] Registration of the crio container factory successfully Aug 04 19:15:55.428231 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.428191 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 19:15:55.428231 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.428213 2580 factory.go:103] Registering Raw factory Aug 04 19:15:55.428231 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.428228 2580 manager.go:1196] Started watching for new ooms in manager Aug 04 19:15:55.428816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.428795 2580 manager.go:349] Starting recovery of all containers Aug 04 19:15:55.434599 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.434574 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bl6sl" Aug 04 19:15:55.435713 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.435685 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 19:15:55.435806 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.435728 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 19:15:55.442844 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.442736 2580 manager.go:354] Recovery completed Aug 04 19:15:55.444048 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.444025 2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Aug 04 19:15:55.446910 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.446899 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:55.448794 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.448778 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:55.448849 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.448806 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:55.448849 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.448816 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:55.449310 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.449298 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 19:15:55.449372 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.449310 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 19:15:55.449372 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.449325 2580 state_mem.go:36] "Initialized new in-memory state store" Aug 04 19:15:55.449372 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.449349 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:55.450668 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.450645 2580 policy_none.go:49] "None policy: Start" Aug 04 19:15:55.450740 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.450676 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 19:15:55.450740 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.450715 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 19:15:55.450845 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.450715 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-99.ec2.internal.18c8b02c0a367b49 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-99.ec2.internal,UID:ip-10-0-136-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-99.ec2.internal,},FirstTimestamp:2026-08-04 19:15:55.448793929 +0000 UTC m=+0.354394458,LastTimestamp:2026-08-04 19:15:55.448793929 +0000 UTC m=+0.354394458,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-99.ec2.internal,}" Aug 04 19:15:55.450845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.450784 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:55.450845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.450815 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:55.450845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.450831 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:55.451864 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.451853 2580 policy_none.go:47] "Start" Aug 04 19:15:55.492050 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.492032 2580 manager.go:337] "Starting Device Plugin manager" Aug 04 19:15:55.492133 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.492060 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 19:15:55.492133 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.492070 2580 server.go:87] "Starting device plugin registration server" Aug 04 19:15:55.492307 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.492294 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 19:15:55.492350 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.492310 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 19:15:55.492415 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.492395 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 19:15:55.492531 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.492514 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 19:15:55.492531 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.492529 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 19:15:55.492945 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.492927 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 19:15:55.493017 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.492973 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:55.593049 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.593016 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:55.594068 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.593927 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:55.594068 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.593958 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:55.594068 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.593968 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:55.594068 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.593995 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.595488 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.595454 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 19:15:55.596931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.596914 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 19:15:55.597010 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.596948 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 19:15:55.597010 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.596966 2580 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 19:15:55.597010 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.597001 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 19:15:55.599222 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.599173 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:55.602184 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.602170 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.602247 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.602190 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-99.ec2.internal\": node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:55.641966 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.641942 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:55.697635 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.697600 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal"] Aug 04 19:15:55.697733 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.697686 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:55.699152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.699137 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:55.699230 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.699169 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:55.699230 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.699181 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:55.700292 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.700278 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:55.700399 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.700376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.700438 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.700404 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:55.700970 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.700955 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:55.700970 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.700982 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:55.701112 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.701000 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:55.701112 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.700998 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:55.701112 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.701085 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:55.701112 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.701101 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:55.702233 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.702218 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.702278 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.702250 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:55.702886 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.702871 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:55.702975 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.702901 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:55.702975 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.702915 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:55.726117 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.726095 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-99.ec2.internal\" not found" node="ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.730458 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.730436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c1b597b5dbb0bee34fd8da372560306-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"3c1b597b5dbb0bee34fd8da372560306\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.730537 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.730484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c1b597b5dbb0bee34fd8da372560306-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"3c1b597b5dbb0bee34fd8da372560306\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.730582 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.730486 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-99.ec2.internal\" not found" node="ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.730875 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.730863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f3b2eaee104600060fbf45b71994dca-config\") pod \"kube-apiserver-proxy-ip-10-0-136-99.ec2.internal\" (UID: \"2f3b2eaee104600060fbf45b71994dca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.742503 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.742489 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:55.831805 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.831770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c1b597b5dbb0bee34fd8da372560306-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"3c1b597b5dbb0bee34fd8da372560306\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.831805 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.831806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c1b597b5dbb0bee34fd8da372560306-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"3c1b597b5dbb0bee34fd8da372560306\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.831936 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.831825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f3b2eaee104600060fbf45b71994dca-config\") pod \"kube-apiserver-proxy-ip-10-0-136-99.ec2.internal\" (UID: \"2f3b2eaee104600060fbf45b71994dca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.831936 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.831885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c1b597b5dbb0bee34fd8da372560306-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"3c1b597b5dbb0bee34fd8da372560306\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.832004 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.831941 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c1b597b5dbb0bee34fd8da372560306-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"3c1b597b5dbb0bee34fd8da372560306\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.832004 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:55.831974 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f3b2eaee104600060fbf45b71994dca-config\") pod \"kube-apiserver-proxy-ip-10-0-136-99.ec2.internal\" (UID: \"2f3b2eaee104600060fbf45b71994dca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" Aug 04 19:15:55.842876 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.842858 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:55.943742 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:55.943693 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:56.029043 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.029007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" Aug 04 19:15:56.034309 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.034294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" Aug 04 19:15:56.043794 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:56.043777 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:56.144256 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:56.144229 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:56.244814 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:56.244760 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:56.345374 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:56.345350 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:56.372513 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.372493 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 19:15:56.372877 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.372619 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 19:15:56.393580 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.393560 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:56.445593 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.445562 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 19:10:56 +0000 UTC" deadline="2028-01-09 18:30:25.152560647 +0000 UTC" Aug 04 19:15:56.445698 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.445598 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12551h14m28.706967412s" Aug 04 19:15:56.445698 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:56.445572 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:56.461080 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.461063 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:56.546118 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:56.546090 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found" Aug 04 19:15:56.554166 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.554147 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:56.554428 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:56.554406 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3b2eaee104600060fbf45b71994dca.slice/crio-66a862adec4e4c877eb347909f6bc9de3cd1d560b8ab58a25b300f0860d7c819 WatchSource:0}: Error finding container 66a862adec4e4c877eb347909f6bc9de3cd1d560b8ab58a25b300f0860d7c819: Status 404 returned error can't find the container with id 66a862adec4e4c877eb347909f6bc9de3cd1d560b8ab58a25b300f0860d7c819 Aug 04 19:15:56.554833 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:56.554812 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1b597b5dbb0bee34fd8da372560306.slice/crio-e970255cfcbef8a8a07f477d86dd102d303ac5a3830f8cab9502b2bcb5e26c4d WatchSource:0}: Error finding container e970255cfcbef8a8a07f477d86dd102d303ac5a3830f8cab9502b2bcb5e26c4d: Status 404 returned error can't find the container with id e970255cfcbef8a8a07f477d86dd102d303ac5a3830f8cab9502b2bcb5e26c4d Aug 04 19:15:56.561003 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.560988 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:15:56.600221 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.600178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" event={"ID":"3c1b597b5dbb0bee34fd8da372560306","Type":"ContainerStarted","Data":"e970255cfcbef8a8a07f477d86dd102d303ac5a3830f8cab9502b2bcb5e26c4d"} Aug 04 19:15:56.601260 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.601239 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" event={"ID":"2f3b2eaee104600060fbf45b71994dca","Type":"ContainerStarted","Data":"66a862adec4e4c877eb347909f6bc9de3cd1d560b8ab58a25b300f0860d7c819"} Aug 04 19:15:56.627335 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.627301 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" Aug 04 19:15:56.641582 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.641562 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 19:15:56.642562 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.642529 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" Aug 04 19:15:56.656454 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:56.656440 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 19:15:57.411666 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.411639 2580 apiserver.go:52] "Watching apiserver" Aug 04 19:15:57.424013 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.423981 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 19:15:57.424547 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.424497 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-t9kwv","openshift-network-operator/iptables-alerter-bbwn4","kube-system/konnectivity-agent-44vkh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq","openshift-dns/node-resolver-gk8cr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal","openshift-multus/multus-additional-cni-plugins-94vkf","openshift-network-diagnostics/network-check-target-wk54h","openshift-ovn-kubernetes/ovnkube-node-j26l2","kube-system/global-pull-secret-syncer-r9jp7","kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jzxzt","openshift-image-registry/node-ca-jvzq8","openshift-multus/multus-bk6zz"] Aug 04 19:15:57.426226 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.426193 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.427509 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.427470 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 19:15:57.427945 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.427923 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.429976 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.429956 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 19:15:57.430332 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.430312 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 19:15:57.430519 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.430500 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 19:15:57.430720 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.430698 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.430960 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.430940 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.431204 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.431185 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hrqkv\"" Aug 04 19:15:57.431799 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.431711 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.431799 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.431731 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 19:15:57.431941 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.431716 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lwvdr\"" Aug 04 19:15:57.431941 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.431936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:15:57.432382 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.432360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.433270 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.432777 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.438906 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.438888 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:15:57.439004 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.438945 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.439004 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.438957 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:15:57.439004 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.438993 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.439170 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.439048 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.439170 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.439150 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 19:15:57.439170 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.439153 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4s5dp\"" Aug 04 19:15:57.439326 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.439248 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.439382 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.439327 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 19:15:57.440468 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.440446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.440839 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.440818 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 19:15:57.440930 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.440863 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.441074 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-system-cni-dir\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.441142 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-os-release\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.441142 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b803896-90de-4732-bee5-3c7a93aa215e-cni-binary-copy\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.441248 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.441248 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441170 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b803896-90de-4732-bee5-3c7a93aa215e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.441248 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-cnibin\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.441248 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b803896-90de-4732-bee5-3c7a93aa215e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.441403 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmsvb\" (UniqueName: \"kubernetes.io/projected/3b803896-90de-4732-bee5-3c7a93aa215e-kube-api-access-lmsvb\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.441403 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14c3bacb-2797-4096-ace0-d6e5b61e26be-host\") pod \"node-ca-jvzq8\" (UID: \"14c3bacb-2797-4096-ace0-d6e5b61e26be\") " pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.441403 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14c3bacb-2797-4096-ace0-d6e5b61e26be-serviceca\") pod \"node-ca-jvzq8\" (UID: \"14c3bacb-2797-4096-ace0-d6e5b61e26be\") " pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.441403 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.441390 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lc8tl\"" Aug 04 19:15:57.442513 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.442489 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 19:15:57.442839 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.442818 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.442839 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.442832 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.442988 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.442845 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-swlrh\"" Aug 04 19:15:57.443047 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.443015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:57.443142 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.443102 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:15:57.443226 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.443187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf926\" (UniqueName: \"kubernetes.io/projected/14c3bacb-2797-4096-ace0-d6e5b61e26be-kube-api-access-nf926\") pod \"node-ca-jvzq8\" (UID: \"14c3bacb-2797-4096-ace0-d6e5b61e26be\") " pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.443320 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.443244 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.443481 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.443465 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:15:57.443599 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.443530 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:15:57.444353 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444120 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 19:15:57.444353 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444184 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zn78z\"" Aug 04 19:15:57.444353 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444190 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.444585 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444411 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 19:15:57.444787 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444602 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ntkm8\"" Aug 04 19:15:57.444787 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444728 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.445138 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444780 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p7kpm\"" Aug 04 19:15:57.445216 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444885 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 19:15:57.445272 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444929 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.445272 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.445611 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444984 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 19:15:57.445611 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.444988 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 19:15:57.447511 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.447460 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 19:15:57.447872 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.447854 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.447990 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.447955 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.448087 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.448072 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jmh2g\"" Aug 04 19:15:57.466630 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.466608 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ldzhx" Aug 04 19:15:57.473209 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.473189 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ldzhx" Aug 04 19:15:57.528825 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.528802 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 19:15:57.545853 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.545808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46clc\" (UniqueName: \"kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc\") pod \"network-check-target-wk54h\" (UID: \"3747b048-0cd0-42b4-994f-4df2da8e8641\") " pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:15:57.545985 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.545875 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.545985 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.545899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c261189f-7399-4ff4-a82c-06f9f682305d-tmp-dir\") pod \"node-resolver-gk8cr\" (UID: \"c261189f-7399-4ff4-a82c-06f9f682305d\") " pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.545985 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.545917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-kubelet\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.545985 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.545969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-log-socket\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.546183 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.546183 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-socket-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.546183 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-registration-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.546183 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-system-cni-dir\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.546183 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzp7q\" (UniqueName: \"kubernetes.io/projected/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-kube-api-access-gzp7q\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.546341 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-modprobe-d\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.546341 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3542398b-31b9-40e0-a95c-ef5369ed358e-tmp\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.546341 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546241 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-run-k8s-cni-cncf-io\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.546341 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-hostroot\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.546341 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-etc-kubernetes\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.546495 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-sys-fs\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.546495 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-run-systemd\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.546495 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:15:57.546495 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-var-lib-kubelet\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.546495 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-tuned\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.546688 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-run-netns\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.546688 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81435689-dc7b-4c48-b8ec-16034b424424-ovnkube-config\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.546688 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmsvb\" (UniqueName: \"kubernetes.io/projected/3b803896-90de-4732-bee5-3c7a93aa215e-kube-api-access-lmsvb\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.546688 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b8e7d88-add5-4eed-be81-45f84784c12e-iptables-alerter-script\") pod \"iptables-alerter-bbwn4\" (UID: \"9b8e7d88-add5-4eed-be81-45f84784c12e\") " pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.546688 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81435689-dc7b-4c48-b8ec-16034b424424-env-overrides\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.546846 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/241e4560-d288-44ae-af3a-49be302f4d6a-dbus\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:57.546846 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-kubernetes\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.546846 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-systemd\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.546846 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-run-ovn-kubernetes\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.546846 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-sysctl-d\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.547005 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-sys\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.547005 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-device-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.547005 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.546975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-var-lib-openvswitch\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.547005 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-etc-openvswitch\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.547133 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/241e4560-d288-44ae-af3a-49be302f4d6a-kubelet-config\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:57.547133 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-os-release\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.547133 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-cnibin\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.547133 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-socket-dir-parent\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.547266 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-cni-bin\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.547266 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzlc\" (UniqueName: \"kubernetes.io/projected/81435689-dc7b-4c48-b8ec-16034b424424-kube-api-access-qjzlc\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.547266 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29249092-ebd1-4152-b8de-a36c22b99635-konnectivity-ca\") pod \"konnectivity-agent-44vkh\" (UID: \"29249092-ebd1-4152-b8de-a36c22b99635\") " pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:15:57.547373 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-var-lib-cni-multus\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.547373 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-conf-dir\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.547373 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-daemon-config\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.547471 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547389 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-run-multus-certs\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.547471 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-lib-modules\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.547471 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-host\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.547471 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14c3bacb-2797-4096-ace0-d6e5b61e26be-serviceca\") pod \"node-ca-jvzq8\" (UID: \"14c3bacb-2797-4096-ace0-d6e5b61e26be\") " pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.547703 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-os-release\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.547703 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-cni-binary-copy\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.547703 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-systemd-units\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.547703 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-run-ovn\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.547870 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-node-log\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.547870 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-cni-netd\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.547870 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b803896-90de-4732-bee5-3c7a93aa215e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.547870 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-run-openvswitch\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.548005 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln27f\" (UniqueName: \"kubernetes.io/projected/3542398b-31b9-40e0-a95c-ef5369ed358e-kube-api-access-ln27f\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.548005 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14c3bacb-2797-4096-ace0-d6e5b61e26be-host\") pod \"node-ca-jvzq8\" (UID: \"14c3bacb-2797-4096-ace0-d6e5b61e26be\") " pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.548005 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.547978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf926\" (UniqueName: \"kubernetes.io/projected/14c3bacb-2797-4096-ace0-d6e5b61e26be-kube-api-access-nf926\") pod \"node-ca-jvzq8\" (UID: \"14c3bacb-2797-4096-ace0-d6e5b61e26be\") " pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.548106 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.548017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b803896-90de-4732-bee5-3c7a93aa215e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.548106 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.548016 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-os-release\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.548106 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.548057 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-sysctl-conf\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.548106 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.548095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-run\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.548239 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.548127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-cnibin\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.548239 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.548200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-cnibin\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.548239 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.548183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8tb8\" (UniqueName: \"kubernetes.io/projected/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-kube-api-access-x8tb8\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.548446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.548417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-system-cni-dir\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.548562 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.548453 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14c3bacb-2797-4096-ace0-d6e5b61e26be-host\") pod \"node-ca-jvzq8\" (UID: \"14c3bacb-2797-4096-ace0-d6e5b61e26be\") " pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.549014 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.548992 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14c3bacb-2797-4096-ace0-d6e5b61e26be-serviceca\") pod \"node-ca-jvzq8\" (UID: \"14c3bacb-2797-4096-ace0-d6e5b61e26be\") " pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.549125 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c261189f-7399-4ff4-a82c-06f9f682305d-hosts-file\") pod \"node-resolver-gk8cr\" (UID: \"c261189f-7399-4ff4-a82c-06f9f682305d\") " pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.549191 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b803896-90de-4732-bee5-3c7a93aa215e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.549323 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-slash\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.549394 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81435689-dc7b-4c48-b8ec-16034b424424-ovn-node-metrics-cert\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.549451 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7clg\" (UniqueName: \"kubernetes.io/projected/4e6ee10d-1bcc-42fc-911d-76886bc6e292-kube-api-access-z7clg\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:15:57.549535 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-run-netns\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.549535 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549501 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b803896-90de-4732-bee5-3c7a93aa215e-cni-binary-copy\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.549535 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b803896-90de-4732-bee5-3c7a93aa215e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.549736 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.549736 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549671 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b803896-90de-4732-bee5-3c7a93aa215e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.549736 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-cni-dir\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.549736 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b8e7d88-add5-4eed-be81-45f84784c12e-host-slash\") pod \"iptables-alerter-bbwn4\" (UID: \"9b8e7d88-add5-4eed-be81-45f84784c12e\") " pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.549736 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.549987 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k5rz\" (UniqueName: \"kubernetes.io/projected/c261189f-7399-4ff4-a82c-06f9f682305d-kube-api-access-6k5rz\") pod \"node-resolver-gk8cr\" (UID: \"c261189f-7399-4ff4-a82c-06f9f682305d\") " pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.549987 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29249092-ebd1-4152-b8de-a36c22b99635-agent-certs\") pod \"konnectivity-agent-44vkh\" (UID: \"29249092-ebd1-4152-b8de-a36c22b99635\") " pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:15:57.549987 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-system-cni-dir\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.549987 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-var-lib-cni-bin\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.550218 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.549993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-var-lib-kubelet\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.550218 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.550026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xrhj\" (UniqueName: \"kubernetes.io/projected/9b8e7d88-add5-4eed-be81-45f84784c12e-kube-api-access-8xrhj\") pod \"iptables-alerter-bbwn4\" (UID: \"9b8e7d88-add5-4eed-be81-45f84784c12e\") " pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.550218 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.550061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81435689-dc7b-4c48-b8ec-16034b424424-ovnkube-script-lib\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.550218 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.550073 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b803896-90de-4732-bee5-3c7a93aa215e-cni-binary-copy\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.550218 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.550094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:57.550218 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.550146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-sysconfig\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.554974 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.554951 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:57.561201 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.561177 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 19:15:57.565697 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.565675 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf926\" (UniqueName: \"kubernetes.io/projected/14c3bacb-2797-4096-ace0-d6e5b61e26be-kube-api-access-nf926\") pod \"node-ca-jvzq8\" (UID: \"14c3bacb-2797-4096-ace0-d6e5b61e26be\") " pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.565809 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.565782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmsvb\" (UniqueName: \"kubernetes.io/projected/3b803896-90de-4732-bee5-3c7a93aa215e-kube-api-access-lmsvb\") pod \"multus-additional-cni-plugins-94vkf\" (UID: \"3b803896-90de-4732-bee5-3c7a93aa215e\") " pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.651109 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8tb8\" (UniqueName: \"kubernetes.io/projected/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-kube-api-access-x8tb8\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.651109 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c261189f-7399-4ff4-a82c-06f9f682305d-hosts-file\") pod \"node-resolver-gk8cr\" (UID: \"c261189f-7399-4ff4-a82c-06f9f682305d\") " pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.651314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-slash\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.651314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81435689-dc7b-4c48-b8ec-16034b424424-ovn-node-metrics-cert\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.651314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7clg\" (UniqueName: \"kubernetes.io/projected/4e6ee10d-1bcc-42fc-911d-76886bc6e292-kube-api-access-z7clg\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:15:57.651314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-run-netns\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.651314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651229 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c261189f-7399-4ff4-a82c-06f9f682305d-hosts-file\") pod \"node-resolver-gk8cr\" (UID: \"c261189f-7399-4ff4-a82c-06f9f682305d\") " pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.651314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-cni-dir\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.651314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-run-netns\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.651314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b8e7d88-add5-4eed-be81-45f84784c12e-host-slash\") pod \"iptables-alerter-bbwn4\" (UID: \"9b8e7d88-add5-4eed-be81-45f84784c12e\") " pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k5rz\" (UniqueName: \"kubernetes.io/projected/c261189f-7399-4ff4-a82c-06f9f682305d-kube-api-access-6k5rz\") pod \"node-resolver-gk8cr\" (UID: \"c261189f-7399-4ff4-a82c-06f9f682305d\") " pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29249092-ebd1-4152-b8de-a36c22b99635-agent-certs\") pod \"konnectivity-agent-44vkh\" (UID: \"29249092-ebd1-4152-b8de-a36c22b99635\") " pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b8e7d88-add5-4eed-be81-45f84784c12e-host-slash\") pod \"iptables-alerter-bbwn4\" (UID: \"9b8e7d88-add5-4eed-be81-45f84784c12e\") " pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651400 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-cni-dir\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651212 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-slash\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-system-cni-dir\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-var-lib-cni-bin\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-system-cni-dir\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-var-lib-kubelet\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xrhj\" (UniqueName: \"kubernetes.io/projected/9b8e7d88-add5-4eed-be81-45f84784c12e-kube-api-access-8xrhj\") pod \"iptables-alerter-bbwn4\" (UID: \"9b8e7d88-add5-4eed-be81-45f84784c12e\") " pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-var-lib-cni-bin\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81435689-dc7b-4c48-b8ec-16034b424424-ovnkube-script-lib\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-var-lib-kubelet\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-sysconfig\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.651770 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.651669 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46clc\" (UniqueName: \"kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc\") pod \"network-check-target-wk54h\" (UID: \"3747b048-0cd0-42b4-994f-4df2da8e8641\") " pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c261189f-7399-4ff4-a82c-06f9f682305d-tmp-dir\") pod \"node-resolver-gk8cr\" (UID: \"c261189f-7399-4ff4-a82c-06f9f682305d\") " pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651828 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-sysconfig\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-kubelet\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-log-socket\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-socket-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.651973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-registration-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzp7q\" (UniqueName: \"kubernetes.io/projected/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-kube-api-access-gzp7q\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-modprobe-d\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3542398b-31b9-40e0-a95c-ef5369ed358e-tmp\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-run-k8s-cni-cncf-io\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-hostroot\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-etc-kubernetes\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652224 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-sys-fs\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.652679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-run-systemd\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-socket-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-var-lib-kubelet\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-tuned\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652369 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81435689-dc7b-4c48-b8ec-16034b424424-ovnkube-script-lib\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-run-netns\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81435689-dc7b-4c48-b8ec-16034b424424-ovnkube-config\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652442 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-hostroot\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.652506 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret podName:241e4560-d288-44ae-af3a-49be302f4d6a nodeName:}" failed. No retries permitted until 2026-08-04 19:15:58.15246063 +0000 UTC m=+3.058061151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret") pod "global-pull-secret-syncer-r9jp7" (UID: "241e4560-d288-44ae-af3a-49be302f4d6a") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-run-netns\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-log-socket\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-kubelet\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c261189f-7399-4ff4-a82c-06f9f682305d-tmp-dir\") pod \"node-resolver-gk8cr\" (UID: \"c261189f-7399-4ff4-a82c-06f9f682305d\") " pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652639 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-run-systemd\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652664 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-etc-kubernetes\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.653446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652703 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-sys-fs\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652748 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-var-lib-kubelet\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.652783 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-modprobe-d\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.652849 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs podName:4e6ee10d-1bcc-42fc-911d-76886bc6e292 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:58.15282818 +0000 UTC m=+3.058428720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs") pod "network-metrics-daemon-t9kwv" (UID: "4e6ee10d-1bcc-42fc-911d-76886bc6e292") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652891 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-registration-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-run-k8s-cni-cncf-io\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b8e7d88-add5-4eed-be81-45f84784c12e-iptables-alerter-script\") pod \"iptables-alerter-bbwn4\" (UID: \"9b8e7d88-add5-4eed-be81-45f84784c12e\") " pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81435689-dc7b-4c48-b8ec-16034b424424-env-overrides\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.652997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b8e7d88-add5-4eed-be81-45f84784c12e-iptables-alerter-script\") pod \"iptables-alerter-bbwn4\" (UID: \"9b8e7d88-add5-4eed-be81-45f84784c12e\") " pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/241e4560-d288-44ae-af3a-49be302f4d6a-dbus\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-kubernetes\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653043 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81435689-dc7b-4c48-b8ec-16034b424424-ovnkube-config\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-kubernetes\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-systemd\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-run-ovn-kubernetes\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.654359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653166 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-systemd\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/241e4560-d288-44ae-af3a-49be302f4d6a-dbus\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-sysctl-d\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-sys\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-device-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-var-lib-openvswitch\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653306 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-sysctl-d\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-etc-openvswitch\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81435689-dc7b-4c48-b8ec-16034b424424-env-overrides\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/241e4560-d288-44ae-af3a-49be302f4d6a-kubelet-config\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653362 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-run-ovn-kubernetes\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-cnibin\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653391 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-device-dir\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653406 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-sys\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653420 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-etc-openvswitch\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653432 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-cnibin\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/241e4560-d288-44ae-af3a-49be302f4d6a-kubelet-config\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-socket-dir-parent\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.655234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-var-lib-openvswitch\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-socket-dir-parent\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-cni-bin\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-cni-bin\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzlc\" (UniqueName: \"kubernetes.io/projected/81435689-dc7b-4c48-b8ec-16034b424424-kube-api-access-qjzlc\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29249092-ebd1-4152-b8de-a36c22b99635-konnectivity-ca\") pod \"konnectivity-agent-44vkh\" (UID: \"29249092-ebd1-4152-b8de-a36c22b99635\") " pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-var-lib-cni-multus\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-conf-dir\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-daemon-config\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-var-lib-cni-multus\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653919 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-conf-dir\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-run-multus-certs\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-host-run-multus-certs\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-lib-modules\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.653997 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-host\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-os-release\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-cni-binary-copy\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-systemd-units\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656042 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-run-ovn\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-node-log\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-cni-netd\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-run-openvswitch\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln27f\" (UniqueName: \"kubernetes.io/projected/3542398b-31b9-40e0-a95c-ef5369ed358e-kube-api-access-ln27f\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-sysctl-conf\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-run\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654273 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-systemd-units\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654342 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29249092-ebd1-4152-b8de-a36c22b99635-konnectivity-ca\") pod \"konnectivity-agent-44vkh\" (UID: \"29249092-ebd1-4152-b8de-a36c22b99635\") " pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-run\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654395 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-run-ovn\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-lib-modules\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654425 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-run-openvswitch\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-host\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-host-cni-netd\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81435689-dc7b-4c48-b8ec-16034b424424-node-log\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-os-release\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29249092-ebd1-4152-b8de-a36c22b99635-agent-certs\") pod \"konnectivity-agent-44vkh\" (UID: \"29249092-ebd1-4152-b8de-a36c22b99635\") " pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:15:57.656735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.654745 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-sysctl-conf\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.657383 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.655611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3542398b-31b9-40e0-a95c-ef5369ed358e-etc-tuned\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.657383 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.655629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3542398b-31b9-40e0-a95c-ef5369ed358e-tmp\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.657383 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.655675 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-cni-binary-copy\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.657956 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.657931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-multus-daemon-config\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.660906 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.660882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81435689-dc7b-4c48-b8ec-16034b424424-ovn-node-metrics-cert\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.661829 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.661774 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:57.661829 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.661802 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:57.661829 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.661819 2580 projected.go:196] Error preparing data for projected volume kube-api-access-46clc for pod openshift-network-diagnostics/network-check-target-wk54h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:57.662087 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:57.661888 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc podName:3747b048-0cd0-42b4-994f-4df2da8e8641 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:58.161864799 +0000 UTC m=+3.067465340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-46clc" (UniqueName: "kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc") pod "network-check-target-wk54h" (UID: "3747b048-0cd0-42b4-994f-4df2da8e8641") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:57.664451 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.664431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzp7q\" (UniqueName: \"kubernetes.io/projected/85c12473-ac7d-4f24-98e8-bbc0ae6990d2-kube-api-access-gzp7q\") pod \"multus-bk6zz\" (UID: \"85c12473-ac7d-4f24-98e8-bbc0ae6990d2\") " pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.665866 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.665780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xrhj\" (UniqueName: \"kubernetes.io/projected/9b8e7d88-add5-4eed-be81-45f84784c12e-kube-api-access-8xrhj\") pod \"iptables-alerter-bbwn4\" (UID: \"9b8e7d88-add5-4eed-be81-45f84784c12e\") " pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.665866 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.665815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzlc\" (UniqueName: \"kubernetes.io/projected/81435689-dc7b-4c48-b8ec-16034b424424-kube-api-access-qjzlc\") pod \"ovnkube-node-j26l2\" (UID: \"81435689-dc7b-4c48-b8ec-16034b424424\") " pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.665866 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.665848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8tb8\" (UniqueName: \"kubernetes.io/projected/f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6-kube-api-access-x8tb8\") pod \"aws-ebs-csi-driver-node-246vq\" (UID: \"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.667148 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.667127 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7clg\" (UniqueName: \"kubernetes.io/projected/4e6ee10d-1bcc-42fc-911d-76886bc6e292-kube-api-access-z7clg\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:15:57.667231 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.667144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k5rz\" (UniqueName: \"kubernetes.io/projected/c261189f-7399-4ff4-a82c-06f9f682305d-kube-api-access-6k5rz\") pod \"node-resolver-gk8cr\" (UID: \"c261189f-7399-4ff4-a82c-06f9f682305d\") " pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.668145 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.668128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln27f\" (UniqueName: \"kubernetes.io/projected/3542398b-31b9-40e0-a95c-ef5369ed358e-kube-api-access-ln27f\") pod \"tuned-jzxzt\" (UID: \"3542398b-31b9-40e0-a95c-ef5369ed358e\") " pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:57.755145 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.755113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94vkf" Aug 04 19:15:57.764537 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.764512 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvzq8" Aug 04 19:15:57.774465 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.774444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:15:57.780360 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.780340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" Aug 04 19:15:57.791024 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.791003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bk6zz" Aug 04 19:15:57.797947 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.797926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:15:57.805713 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.805694 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gk8cr" Aug 04 19:15:57.814589 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.814569 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bbwn4" Aug 04 19:15:57.820156 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:57.820133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" Aug 04 19:15:58.158959 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:58.158865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:15:58.158959 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:58.158934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:58.159180 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:58.159060 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:58.159180 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:58.159073 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:58.159180 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:58.159125 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret podName:241e4560-d288-44ae-af3a-49be302f4d6a nodeName:}" failed. No retries permitted until 2026-08-04 19:15:59.159106189 +0000 UTC m=+4.064706714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret") pod "global-pull-secret-syncer-r9jp7" (UID: "241e4560-d288-44ae-af3a-49be302f4d6a") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:58.159180 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:58.159147 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs podName:4e6ee10d-1bcc-42fc-911d-76886bc6e292 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:59.159136649 +0000 UTC m=+4.064737175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs") pod "network-metrics-daemon-t9kwv" (UID: "4e6ee10d-1bcc-42fc-911d-76886bc6e292") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:58.259870 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:58.259832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46clc\" (UniqueName: \"kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc\") pod \"network-check-target-wk54h\" (UID: \"3747b048-0cd0-42b4-994f-4df2da8e8641\") " pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:15:58.260035 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:58.260012 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:58.260093 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:58.260041 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:58.260093 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:58.260053 2580 projected.go:196] Error preparing data for projected volume kube-api-access-46clc for pod openshift-network-diagnostics/network-check-target-wk54h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:58.260163 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:58.260115 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc podName:3747b048-0cd0-42b4-994f-4df2da8e8641 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:59.260098405 +0000 UTC m=+4.165698924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-46clc" (UniqueName: "kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc") pod "network-check-target-wk54h" (UID: "3747b048-0cd0-42b4-994f-4df2da8e8641") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:58.475074 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:58.474968 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 19:10:57 +0000 UTC" deadline="2028-02-13 09:43:14.404711159 +0000 UTC" Aug 04 19:15:58.475074 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:58.475029 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13382h27m15.929686239s" Aug 04 19:15:58.597744 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:58.597720 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:58.597906 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:58.597821 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:15:58.627706 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:58.627669 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c12473_ac7d_4f24_98e8_bbc0ae6990d2.slice/crio-b51f7c6f03362808b57b5107b58f09a3b138df3771bc398422c77c29dedc7880 WatchSource:0}: Error finding container b51f7c6f03362808b57b5107b58f09a3b138df3771bc398422c77c29dedc7880: Status 404 returned error can't find the container with id b51f7c6f03362808b57b5107b58f09a3b138df3771bc398422c77c29dedc7880 Aug 04 19:15:58.629840 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:58.629684 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c3bacb_2797_4096_ace0_d6e5b61e26be.slice/crio-8af5607e5ef443efd1365863703b6d2dadbf2e9aa20cda8c78e6cc7e360e1993 WatchSource:0}: Error finding container 8af5607e5ef443efd1365863703b6d2dadbf2e9aa20cda8c78e6cc7e360e1993: Status 404 returned error can't find the container with id 8af5607e5ef443efd1365863703b6d2dadbf2e9aa20cda8c78e6cc7e360e1993 Aug 04 19:15:58.633767 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:58.633689 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71a9de5_2fdd_48ae_9fc0_4884b8ca23f6.slice/crio-159edfcd5cd8595f22e9f1525cc914bb1c1b46a213f17745973475201c771b1e WatchSource:0}: Error finding container 159edfcd5cd8595f22e9f1525cc914bb1c1b46a213f17745973475201c771b1e: Status 404 returned error can't find the container with id 159edfcd5cd8595f22e9f1525cc914bb1c1b46a213f17745973475201c771b1e Aug 04 19:15:58.635350 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:58.635241 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29249092_ebd1_4152_b8de_a36c22b99635.slice/crio-867f254dbd25405f282feee6de52ad640d45eb296f6e124a234cda5bed750fe9 WatchSource:0}: Error finding container 867f254dbd25405f282feee6de52ad640d45eb296f6e124a234cda5bed750fe9: Status 404 returned error can't find the container with id 867f254dbd25405f282feee6de52ad640d45eb296f6e124a234cda5bed750fe9 Aug 04 19:15:58.639266 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:58.636763 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3542398b_31b9_40e0_a95c_ef5369ed358e.slice/crio-1b90027b0f36e7f1759fd90786719533faed863aa4e6eb0bb1e1dac94132a47f WatchSource:0}: Error finding container 1b90027b0f36e7f1759fd90786719533faed863aa4e6eb0bb1e1dac94132a47f: Status 404 returned error can't find the container with id 1b90027b0f36e7f1759fd90786719533faed863aa4e6eb0bb1e1dac94132a47f Aug 04 19:15:58.639266 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:58.637964 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8e7d88_add5_4eed_be81_45f84784c12e.slice/crio-bec355ca5a3cc54b8e1c757f15a73cd9ef15a0e4906a050c0e2f5f0ac15ca604 WatchSource:0}: Error finding container bec355ca5a3cc54b8e1c757f15a73cd9ef15a0e4906a050c0e2f5f0ac15ca604: Status 404 returned error can't find the container with id bec355ca5a3cc54b8e1c757f15a73cd9ef15a0e4906a050c0e2f5f0ac15ca604 Aug 04 19:15:58.639266 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:58.638816 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc261189f_7399_4ff4_a82c_06f9f682305d.slice/crio-327ff0adeac9c6e69cc32ca783ecf64b8826190c283ae7d4b732e62d43197be3 WatchSource:0}: Error finding container 327ff0adeac9c6e69cc32ca783ecf64b8826190c283ae7d4b732e62d43197be3: Status 404 returned error can't find the container with id 327ff0adeac9c6e69cc32ca783ecf64b8826190c283ae7d4b732e62d43197be3 Aug 04 19:15:58.641739 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:58.640351 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81435689_dc7b_4c48_b8ec_16034b424424.slice/crio-47a1227569a619b0b767467213b4485e807334028d9f381c3ce6afa3a7c45862 WatchSource:0}: Error finding container 47a1227569a619b0b767467213b4485e807334028d9f381c3ce6afa3a7c45862: Status 404 returned error can't find the container with id 47a1227569a619b0b767467213b4485e807334028d9f381c3ce6afa3a7c45862 Aug 04 19:15:58.641739 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:15:58.641640 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b803896_90de_4732_bee5_3c7a93aa215e.slice/crio-4b2c7553aef678857ba7bdd7a250e10ae2b0fc74fcdd9d21054e3eaee9fc8b93 WatchSource:0}: Error finding container 4b2c7553aef678857ba7bdd7a250e10ae2b0fc74fcdd9d21054e3eaee9fc8b93: Status 404 returned error can't find the container with id 4b2c7553aef678857ba7bdd7a250e10ae2b0fc74fcdd9d21054e3eaee9fc8b93 Aug 04 19:15:59.166845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.166534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:15:59.167017 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:59.166848 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:59.167017 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.166875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:15:59.167017 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:59.166927 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs podName:4e6ee10d-1bcc-42fc-911d-76886bc6e292 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:01.166902891 +0000 UTC m=+6.072503426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs") pod "network-metrics-daemon-t9kwv" (UID: "4e6ee10d-1bcc-42fc-911d-76886bc6e292") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:59.167017 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:59.166988 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:59.167293 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:59.167041 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret podName:241e4560-d288-44ae-af3a-49be302f4d6a nodeName:}" failed. No retries permitted until 2026-08-04 19:16:01.167024406 +0000 UTC m=+6.072624925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret") pod "global-pull-secret-syncer-r9jp7" (UID: "241e4560-d288-44ae-af3a-49be302f4d6a") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:59.268070 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.268028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46clc\" (UniqueName: \"kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc\") pod \"network-check-target-wk54h\" (UID: \"3747b048-0cd0-42b4-994f-4df2da8e8641\") " pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:15:59.268279 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:59.268260 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:59.268349 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:59.268287 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:59.268349 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:59.268301 2580 projected.go:196] Error preparing data for projected volume kube-api-access-46clc for pod openshift-network-diagnostics/network-check-target-wk54h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:59.268471 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:59.268364 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc podName:3747b048-0cd0-42b4-994f-4df2da8e8641 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:01.268344254 +0000 UTC m=+6.173944782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-46clc" (UniqueName: "kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc") pod "network-check-target-wk54h" (UID: "3747b048-0cd0-42b4-994f-4df2da8e8641") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:59.475964 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.475829 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 19:10:57 +0000 UTC" deadline="2028-02-11 14:50:44.67439803 +0000 UTC" Aug 04 19:15:59.475964 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.475867 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13339h34m45.198536326s" Aug 04 19:15:59.600395 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.599652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:15:59.600395 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:59.599786 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:15:59.600395 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.600223 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:15:59.600395 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:15:59.600336 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:15:59.621237 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.621198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" event={"ID":"2f3b2eaee104600060fbf45b71994dca","Type":"ContainerStarted","Data":"b370e000d03cca847a34f18666f1d43f280bde8119fe9ff80e89152c6e885d42"} Aug 04 19:15:59.625818 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.625788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vkf" event={"ID":"3b803896-90de-4732-bee5-3c7a93aa215e","Type":"ContainerStarted","Data":"4b2c7553aef678857ba7bdd7a250e10ae2b0fc74fcdd9d21054e3eaee9fc8b93"} Aug 04 19:15:59.630134 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.630049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" event={"ID":"81435689-dc7b-4c48-b8ec-16034b424424","Type":"ContainerStarted","Data":"47a1227569a619b0b767467213b4485e807334028d9f381c3ce6afa3a7c45862"} Aug 04 19:15:59.635072 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.635028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" event={"ID":"3542398b-31b9-40e0-a95c-ef5369ed358e","Type":"ContainerStarted","Data":"1b90027b0f36e7f1759fd90786719533faed863aa4e6eb0bb1e1dac94132a47f"} Aug 04 19:15:59.639021 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.638750 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" podStartSLOduration=2.63873063 podStartE2EDuration="2.63873063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:59.638337113 +0000 UTC m=+4.543937659" watchObservedRunningTime="2026-08-04 19:15:59.63873063 +0000 UTC m=+4.544331176" Aug 04 19:15:59.640305 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.640235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" event={"ID":"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6","Type":"ContainerStarted","Data":"159edfcd5cd8595f22e9f1525cc914bb1c1b46a213f17745973475201c771b1e"} Aug 04 19:15:59.645776 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.645748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bk6zz" event={"ID":"85c12473-ac7d-4f24-98e8-bbc0ae6990d2","Type":"ContainerStarted","Data":"b51f7c6f03362808b57b5107b58f09a3b138df3771bc398422c77c29dedc7880"} Aug 04 19:15:59.668410 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.668321 2580 generic.go:356] "Generic (PLEG): container finished" podID="3c1b597b5dbb0bee34fd8da372560306" containerID="5fd9e5ccba6bf422006face1a68f605ada248fb760a432a229ff8f622be63b4e" exitCode=0 Aug 04 19:15:59.668584 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.668479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" event={"ID":"3c1b597b5dbb0bee34fd8da372560306","Type":"ContainerDied","Data":"5fd9e5ccba6bf422006face1a68f605ada248fb760a432a229ff8f622be63b4e"} Aug 04 19:15:59.680140 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.680104 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gk8cr" event={"ID":"c261189f-7399-4ff4-a82c-06f9f682305d","Type":"ContainerStarted","Data":"327ff0adeac9c6e69cc32ca783ecf64b8826190c283ae7d4b732e62d43197be3"} Aug 04 19:15:59.697064 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.697016 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bbwn4" event={"ID":"9b8e7d88-add5-4eed-be81-45f84784c12e","Type":"ContainerStarted","Data":"bec355ca5a3cc54b8e1c757f15a73cd9ef15a0e4906a050c0e2f5f0ac15ca604"} Aug 04 19:15:59.708258 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.708221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-44vkh" event={"ID":"29249092-ebd1-4152-b8de-a36c22b99635","Type":"ContainerStarted","Data":"867f254dbd25405f282feee6de52ad640d45eb296f6e124a234cda5bed750fe9"} Aug 04 19:15:59.727604 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:15:59.727426 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvzq8" event={"ID":"14c3bacb-2797-4096-ace0-d6e5b61e26be","Type":"ContainerStarted","Data":"8af5607e5ef443efd1365863703b6d2dadbf2e9aa20cda8c78e6cc7e360e1993"} Aug 04 19:16:00.598527 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:00.598057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:00.598527 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:00.598193 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:00.744506 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:00.744464 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" event={"ID":"3c1b597b5dbb0bee34fd8da372560306","Type":"ContainerStarted","Data":"fc1b5dd5e9fd667f2563c7b34455a6374f35cbebb6eefe2034812ae2c6383d06"} Aug 04 19:16:01.185242 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:01.185195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:01.185432 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:01.185288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:01.185527 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:01.185483 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:16:01.185589 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:01.185578 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret podName:241e4560-d288-44ae-af3a-49be302f4d6a nodeName:}" failed. No retries permitted until 2026-08-04 19:16:05.185557294 +0000 UTC m=+10.091157820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret") pod "global-pull-secret-syncer-r9jp7" (UID: "241e4560-d288-44ae-af3a-49be302f4d6a") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:16:01.186030 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:01.186006 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:16:01.186141 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:01.186073 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs podName:4e6ee10d-1bcc-42fc-911d-76886bc6e292 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:05.186055092 +0000 UTC m=+10.091655624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs") pod "network-metrics-daemon-t9kwv" (UID: "4e6ee10d-1bcc-42fc-911d-76886bc6e292") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:16:01.286699 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:01.286658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46clc\" (UniqueName: \"kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc\") pod \"network-check-target-wk54h\" (UID: \"3747b048-0cd0-42b4-994f-4df2da8e8641\") " pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:01.286903 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:01.286853 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:16:01.286903 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:01.286874 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:16:01.286903 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:01.286889 2580 projected.go:196] Error preparing data for projected volume kube-api-access-46clc for pod openshift-network-diagnostics/network-check-target-wk54h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:16:01.287027 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:01.286951 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc podName:3747b048-0cd0-42b4-994f-4df2da8e8641 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:05.286930617 +0000 UTC m=+10.192531151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-46clc" (UniqueName: "kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc") pod "network-check-target-wk54h" (UID: "3747b048-0cd0-42b4-994f-4df2da8e8641") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:16:01.600240 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:01.600088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:01.600751 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:01.600236 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:01.600751 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:01.600666 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:01.600870 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:01.600778 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:02.598328 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:02.598287 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:02.598542 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:02.598427 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:03.600760 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:03.600236 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:03.600760 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:03.600370 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:03.600760 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:03.600616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:03.600760 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:03.600719 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:04.598074 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:04.597632 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:04.598074 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:04.597750 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:05.223927 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:05.223888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:05.224386 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:05.223958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:05.224386 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:05.224077 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:16:05.224386 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:05.224135 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs podName:4e6ee10d-1bcc-42fc-911d-76886bc6e292 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:13.224115912 +0000 UTC m=+18.129716445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs") pod "network-metrics-daemon-t9kwv" (UID: "4e6ee10d-1bcc-42fc-911d-76886bc6e292") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:16:05.224386 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:05.224195 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:16:05.224386 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:05.224230 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret podName:241e4560-d288-44ae-af3a-49be302f4d6a nodeName:}" failed. No retries permitted until 2026-08-04 19:16:13.224217204 +0000 UTC m=+18.129817727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret") pod "global-pull-secret-syncer-r9jp7" (UID: "241e4560-d288-44ae-af3a-49be302f4d6a") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:16:05.325673 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:05.325059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46clc\" (UniqueName: \"kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc\") pod \"network-check-target-wk54h\" (UID: \"3747b048-0cd0-42b4-994f-4df2da8e8641\") " pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:05.325673 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:05.325235 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:16:05.325673 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:05.325253 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:16:05.325673 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:05.325265 2580 projected.go:196] Error preparing data for projected volume kube-api-access-46clc for pod openshift-network-diagnostics/network-check-target-wk54h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:16:05.325673 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:05.325320 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc podName:3747b048-0cd0-42b4-994f-4df2da8e8641 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:13.325302161 +0000 UTC m=+18.230902693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-46clc" (UniqueName: "kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc") pod "network-check-target-wk54h" (UID: "3747b048-0cd0-42b4-994f-4df2da8e8641") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:16:05.599422 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:05.598994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:05.599422 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:05.599114 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:05.599422 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:05.599160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:05.599422 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:05.599281 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:06.597932 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:06.597899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:06.598371 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:06.598025 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:07.597663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:07.597584 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:07.597827 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:07.597719 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:07.598111 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:07.598085 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:07.598519 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:07.598189 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:08.598520 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:08.597869 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:08.598520 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:08.598030 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:09.597570 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:09.597541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:09.597570 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:09.597568 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:09.597764 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:09.597668 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:09.597823 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:09.597768 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:10.598166 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:10.598130 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:10.598637 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:10.598254 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:11.599292 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:11.599259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:11.599754 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:11.599380 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:11.599817 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:11.599775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:11.599897 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:11.599879 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:12.597771 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:12.597731 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:12.597959 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:12.597855 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:13.288911 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:13.288881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:13.289445 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:13.288934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:13.289445 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:13.289029 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:16:13.289445 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:13.289033 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:16:13.289445 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:13.289095 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs podName:4e6ee10d-1bcc-42fc-911d-76886bc6e292 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:29.289075905 +0000 UTC m=+34.194676423 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs") pod "network-metrics-daemon-t9kwv" (UID: "4e6ee10d-1bcc-42fc-911d-76886bc6e292") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:16:13.289445 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:13.289113 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret podName:241e4560-d288-44ae-af3a-49be302f4d6a nodeName:}" failed. No retries permitted until 2026-08-04 19:16:29.289103756 +0000 UTC m=+34.194704277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret") pod "global-pull-secret-syncer-r9jp7" (UID: "241e4560-d288-44ae-af3a-49be302f4d6a") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:16:13.390093 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:13.390062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46clc\" (UniqueName: \"kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc\") pod \"network-check-target-wk54h\" (UID: \"3747b048-0cd0-42b4-994f-4df2da8e8641\") " pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:13.390240 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:13.390210 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:16:13.390240 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:13.390231 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:16:13.390240 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:13.390239 2580 projected.go:196] Error preparing data for projected volume kube-api-access-46clc for pod openshift-network-diagnostics/network-check-target-wk54h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:16:13.390382 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:13.390294 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc podName:3747b048-0cd0-42b4-994f-4df2da8e8641 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:29.390282482 +0000 UTC m=+34.295883000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-46clc" (UniqueName: "kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc") pod "network-check-target-wk54h" (UID: "3747b048-0cd0-42b4-994f-4df2da8e8641") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:16:13.597631 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:13.597560 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:13.597787 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:13.597704 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:13.597787 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:13.597763 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:13.597918 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:13.597887 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:14.597709 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:14.597681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:14.598118 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:14.597804 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:15.598354 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:15.598318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:15.598815 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:15.598431 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:15.598815 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:15.598763 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:15.598940 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:15.598885 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:16.598017 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:16.597984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:16.598201 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:16.598122 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:17.597559 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.597350 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:17.597559 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:17.597512 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:17.598217 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.598077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:17.598217 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:17.598179 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:17.781414 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.781231 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vkf" event={"ID":"3b803896-90de-4732-bee5-3c7a93aa215e","Type":"ContainerStarted","Data":"1398d032b37ae53aa15d5b7b97ab93387685dbae5b55220c1b406dc4e940cebc"} Aug 04 19:16:17.782644 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.782622 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" event={"ID":"81435689-dc7b-4c48-b8ec-16034b424424","Type":"ContainerStarted","Data":"a026daa2e48e75e3d08c94eaf8cbe9cf27a1c7acc01335e51715f5cd7ea8eeef"} Aug 04 19:16:17.785455 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.785435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" event={"ID":"3542398b-31b9-40e0-a95c-ef5369ed358e","Type":"ContainerStarted","Data":"16611280be0f0b462a375049ddd1f264d7b548d2f6359f5cbcc2541800e8b404"} Aug 04 19:16:17.786994 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.786975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" event={"ID":"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6","Type":"ContainerStarted","Data":"22750633d450d255326d40bd669554badb7c91389632247e8964be5c2324da58"} Aug 04 19:16:17.788249 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.788220 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bk6zz" event={"ID":"85c12473-ac7d-4f24-98e8-bbc0ae6990d2","Type":"ContainerStarted","Data":"bf962f090e05bc03bd514c0413f89806a1dee1ef49c0f95db78a067c907190ac"} Aug 04 19:16:17.789543 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.789523 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gk8cr" event={"ID":"c261189f-7399-4ff4-a82c-06f9f682305d","Type":"ContainerStarted","Data":"da7f2300d6547c72184f2d61d0ea09e0c96b54096133a80274743a18857ee1f8"} Aug 04 19:16:17.790817 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.790799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-44vkh" event={"ID":"29249092-ebd1-4152-b8de-a36c22b99635","Type":"ContainerStarted","Data":"176768b8d2b95dec18425418140d004e4cab19ab178a22ae8b0e7fafb3e05c54"} Aug 04 19:16:17.792009 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.791992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvzq8" event={"ID":"14c3bacb-2797-4096-ace0-d6e5b61e26be","Type":"ContainerStarted","Data":"58589a39ecd6b3ec6011fe306031e3493a7e41179bfb0cb6593006d94b2098fa"} Aug 04 19:16:17.805580 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.805538 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" podStartSLOduration=20.805525154 podStartE2EDuration="20.805525154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:00.760668389 +0000 UTC m=+5.666268935" watchObservedRunningTime="2026-08-04 19:16:17.805525154 +0000 UTC m=+22.711125692" Aug 04 19:16:17.822530 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.822488 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jvzq8" podStartSLOduration=2.997656524 podStartE2EDuration="21.822451223s" totalImagesPullingTime="18.824794699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:17.821984184 +0000 UTC m=+22.727584724" watchObservedRunningTime="2026-08-04 19:16:17.822451223 +0000 UTC m=+22.728051782" Aug 04 19:16:17.844443 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.844402 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-44vkh" podStartSLOduration=3.400521389 podStartE2EDuration="21.84438774s" totalImagesPullingTime="18.443866351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:17.844152213 +0000 UTC m=+22.749752756" watchObservedRunningTime="2026-08-04 19:16:17.84438774 +0000 UTC m=+22.749988280" Aug 04 19:16:17.902774 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.902713 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bk6zz" podStartSLOduration=3.023868244 podStartE2EDuration="21.902691258s" totalImagesPullingTime="18.878823014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:17.873066482 +0000 UTC m=+22.778667021" watchObservedRunningTime="2026-08-04 19:16:17.902691258 +0000 UTC m=+22.808291797" Aug 04 19:16:17.903161 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:17.903127 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gk8cr" podStartSLOduration=3.462885591 podStartE2EDuration="21.90311648s" totalImagesPullingTime="18.440230889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:17.902230916 +0000 UTC m=+22.807831460" watchObservedRunningTime="2026-08-04 19:16:17.90311648 +0000 UTC m=+22.808717020" Aug 04 19:16:18.244249 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.244066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:18.324849 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.324813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:18.494545 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.494503 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:18.595662 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.595635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:18.597539 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.597523 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:18.597646 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:18.597629 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:18.698239 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.698206 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:18.796128 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.796041 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bbwn4" event={"ID":"9b8e7d88-add5-4eed-be81-45f84784c12e","Type":"ContainerStarted","Data":"0bb6709b6f0592fa68eadfd6fd2a7ecc3b16e5f56147092c6737d4c045e3696c"} Aug 04 19:16:18.797416 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.797383 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b803896-90de-4732-bee5-3c7a93aa215e" containerID="1398d032b37ae53aa15d5b7b97ab93387685dbae5b55220c1b406dc4e940cebc" exitCode=0 Aug 04 19:16:18.797554 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.797478 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vkf" event={"ID":"3b803896-90de-4732-bee5-3c7a93aa215e","Type":"ContainerDied","Data":"1398d032b37ae53aa15d5b7b97ab93387685dbae5b55220c1b406dc4e940cebc"} Aug 04 19:16:18.800666 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.800647 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:16:18.801014 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.800993 2580 generic.go:356] "Generic (PLEG): container finished" podID="81435689-dc7b-4c48-b8ec-16034b424424" containerID="3f540d799c9d7f7076e50456ddebcba852e56ac998dd3ca1ad8e5f816fcf22b0" exitCode=1 Aug 04 19:16:18.801126 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.801100 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" event={"ID":"81435689-dc7b-4c48-b8ec-16034b424424","Type":"ContainerStarted","Data":"f4df83d3a79a92922d5a071fbf70eef316bdcd31f2bfff36c9564bc46e967560"} Aug 04 19:16:18.801231 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.801181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" event={"ID":"81435689-dc7b-4c48-b8ec-16034b424424","Type":"ContainerStarted","Data":"5c7f5d86eae04057fb31967bd2fe2c625e8de710e3574911d2e0dfbe1eadaf62"} Aug 04 19:16:18.801288 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.801230 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" event={"ID":"81435689-dc7b-4c48-b8ec-16034b424424","Type":"ContainerStarted","Data":"c052b12197bf9cc4f8189f1291b75be758d37d4946cb88f68639b98054a2c216"} Aug 04 19:16:18.801288 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.801277 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" event={"ID":"81435689-dc7b-4c48-b8ec-16034b424424","Type":"ContainerStarted","Data":"1c6704fded79655bd1151999b5629342712683479b06e77a04502ceaedebc1eb"} Aug 04 19:16:18.801411 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.801322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" event={"ID":"81435689-dc7b-4c48-b8ec-16034b424424","Type":"ContainerDied","Data":"3f540d799c9d7f7076e50456ddebcba852e56ac998dd3ca1ad8e5f816fcf22b0"} Aug 04 19:16:18.814864 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.814821 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jzxzt" podStartSLOduration=3.995600022 podStartE2EDuration="22.814807798s" totalImagesPullingTime="18.819207776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:17.923924746 +0000 UTC m=+22.829525286" watchObservedRunningTime="2026-08-04 19:16:18.814807798 +0000 UTC m=+23.720408368" Aug 04 19:16:18.815041 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.815021 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bbwn4" podStartSLOduration=4.014072927 podStartE2EDuration="22.815017723s" totalImagesPullingTime="18.800944796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:18.814343922 +0000 UTC m=+23.719944462" watchObservedRunningTime="2026-08-04 19:16:18.815017723 +0000 UTC m=+23.720618261" Aug 04 19:16:18.876090 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.876063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:18.976982 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:18.976952 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:19.144378 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:19.144356 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 19:16:19.182510 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:19.182474 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:19.484532 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:19.484424 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:19.507594 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:19.507508 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T19:16:19.144372598Z","UUID":"b237ea04-395d-4048-b5ac-e26026c00306","Handler":null,"Name":"","Endpoint":""} Aug 04 19:16:19.508928 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:19.508899 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 19:16:19.509027 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:19.508940 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 19:16:19.598177 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:19.598142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:19.598661 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:19.598142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:19.598661 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:19.598275 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:19.598661 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:19.598331 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:19.804715 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:19.804627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" event={"ID":"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6","Type":"ContainerStarted","Data":"6ff1fe21efb2c2d6e362c5324fb6ae54f9364a8d629c502c651b6c6d2be36c41"} Aug 04 19:16:20.597525 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:20.597449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:20.597657 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:20.597568 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:20.809881 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:20.809852 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:16:20.810274 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:20.810188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" event={"ID":"81435689-dc7b-4c48-b8ec-16034b424424","Type":"ContainerStarted","Data":"74fcdeb05bc44ad34e93ad686591c0a665d6076c2a1529e52ede501eeb5c6060"} Aug 04 19:16:21.211720 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:21.211526 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:21.283755 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:21.283710 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:21.590628 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:21.590597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:21.597217 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:21.597198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:21.597333 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:21.597313 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:21.597380 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:21.597362 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:21.597495 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:21.597458 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:21.793211 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:21.793095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:21.815184 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:21.815151 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" event={"ID":"f71a9de5-2fdd-48ae-9fc0-4884b8ca23f6","Type":"ContainerStarted","Data":"ea0e74c4c91a77f532a9c85f503506b3cc63507fea41058d8a40569aff472af8"} Aug 04 19:16:21.836816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:21.836757 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-246vq" podStartSLOduration=7.468233422 podStartE2EDuration="25.836739654s" totalImagesPullingTime="18.368506232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:21.836194046 +0000 UTC m=+26.741794584" watchObservedRunningTime="2026-08-04 19:16:21.836739654 +0000 UTC m=+26.742340193" Aug 04 19:16:22.385983 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.385943 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:22.412480 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.412374 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:16:22.413051 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.413034 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:16:22.598084 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.598054 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:22.598238 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:22.598190 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:22.615845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.615814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:22.718560 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.718447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:22.818185 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.817946 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:22.822811 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.822789 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:16:22.823307 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.823277 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" event={"ID":"81435689-dc7b-4c48-b8ec-16034b424424","Type":"ContainerStarted","Data":"e68f1987bb3464c2714bbaf2f829fa6eae82b751f6bbb46a24aa61fef7e78ada"} Aug 04 19:16:22.823772 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.823748 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:16:22.823967 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.823951 2580 scope.go:117] "RemoveContainer" containerID="3f540d799c9d7f7076e50456ddebcba852e56ac998dd3ca1ad8e5f816fcf22b0" Aug 04 19:16:22.824432 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:22.824403 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-44vkh" Aug 04 19:16:23.598026 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:23.597991 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:23.598224 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:23.598115 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:23.598224 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:23.598177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:23.598348 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:23.598319 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:23.829649 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:23.829569 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:16:23.830041 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:23.830013 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" event={"ID":"81435689-dc7b-4c48-b8ec-16034b424424","Type":"ContainerStarted","Data":"85efe32228d2f4580b548528e49b07f4905967b699bb520254fe0f2a2ad333d3"} Aug 04 19:16:23.830510 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:23.830486 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:16:23.830673 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:23.830572 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:16:23.830673 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:23.830623 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:16:23.848095 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:23.848057 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:16:23.848620 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:23.848605 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:16:23.858636 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:23.858595 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" podStartSLOduration=8.995762547 podStartE2EDuration="27.858583117s" totalImagesPullingTime="18.86282057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:23.858447154 +0000 UTC m=+28.764047693" watchObservedRunningTime="2026-08-04 19:16:23.858583117 +0000 UTC m=+28.764183656" Aug 04 19:16:24.598001 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:24.597969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:24.598202 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:24.598077 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:25.601433 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:25.601352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:25.601866 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:25.601439 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:25.601866 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:25.601531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:25.601866 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:25.601633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:25.835986 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:25.835946 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b803896-90de-4732-bee5-3c7a93aa215e" containerID="1484f5a25dc9a75393ac0aeea475a51f71a572c996de736d674250a3b50f8ef0" exitCode=0 Aug 04 19:16:25.836144 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:25.836031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vkf" event={"ID":"3b803896-90de-4732-bee5-3c7a93aa215e","Type":"ContainerDied","Data":"1484f5a25dc9a75393ac0aeea475a51f71a572c996de736d674250a3b50f8ef0"} Aug 04 19:16:26.597892 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:26.597866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:26.597980 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:26.597963 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:26.674195 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:26.674157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:26.738380 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:26.738309 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:26.839840 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:26.839812 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b803896-90de-4732-bee5-3c7a93aa215e" containerID="00d938b6d06b3172caead242267176b63888f9a4aa424b3d9ef3acf30ed80b28" exitCode=0 Aug 04 19:16:26.839840 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:26.839841 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vkf" event={"ID":"3b803896-90de-4732-bee5-3c7a93aa215e","Type":"ContainerDied","Data":"00d938b6d06b3172caead242267176b63888f9a4aa424b3d9ef3acf30ed80b28"} Aug 04 19:16:27.598037 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:27.597625 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:27.598037 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:27.597655 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:27.598037 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:27.597761 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:27.598037 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:27.597781 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:27.844184 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:27.844101 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b803896-90de-4732-bee5-3c7a93aa215e" containerID="7e7a323604149d772e31c392776ea753a3db38b39523536b72ebe49d9a9239dc" exitCode=0 Aug 04 19:16:27.844184 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:27.844152 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vkf" event={"ID":"3b803896-90de-4732-bee5-3c7a93aa215e","Type":"ContainerDied","Data":"7e7a323604149d772e31c392776ea753a3db38b39523536b72ebe49d9a9239dc"} Aug 04 19:16:28.597162 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:28.597131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:28.597359 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:28.597241 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:29.314809 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:29.314776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:29.315259 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:29.314822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:29.315259 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:29.314906 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:16:29.315259 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:29.314908 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:16:29.315259 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:29.314954 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs podName:4e6ee10d-1bcc-42fc-911d-76886bc6e292 nodeName:}" failed. No retries permitted until 2026-08-04 19:17:01.314939859 +0000 UTC m=+66.220540376 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs") pod "network-metrics-daemon-t9kwv" (UID: "4e6ee10d-1bcc-42fc-911d-76886bc6e292") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:16:29.315259 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:29.314967 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret podName:241e4560-d288-44ae-af3a-49be302f4d6a nodeName:}" failed. No retries permitted until 2026-08-04 19:17:01.314960143 +0000 UTC m=+66.220560660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret") pod "global-pull-secret-syncer-r9jp7" (UID: "241e4560-d288-44ae-af3a-49be302f4d6a") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:16:29.415553 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:29.415526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46clc\" (UniqueName: \"kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc\") pod \"network-check-target-wk54h\" (UID: \"3747b048-0cd0-42b4-994f-4df2da8e8641\") " pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:29.415678 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:29.415670 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:16:29.415716 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:29.415687 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:16:29.415716 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:29.415696 2580 projected.go:196] Error preparing data for projected volume kube-api-access-46clc for pod openshift-network-diagnostics/network-check-target-wk54h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:16:29.415778 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:29.415740 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc podName:3747b048-0cd0-42b4-994f-4df2da8e8641 nodeName:}" failed. No retries permitted until 2026-08-04 19:17:01.415726562 +0000 UTC m=+66.321327078 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-46clc" (UniqueName: "kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc") pod "network-check-target-wk54h" (UID: "3747b048-0cd0-42b4-994f-4df2da8e8641") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:16:29.597496 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:29.597411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:29.597640 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:29.597535 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:29.597640 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:29.597618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:29.597754 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:29.597698 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:30.088270 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:30.088129 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:30.180097 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:30.180056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:30.279244 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:30.279218 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:30.378065 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:30.377987 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:30.473404 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:30.473370 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:30.553243 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:30.553218 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:30.598069 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:30.598047 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:30.598231 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:30.598142 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:30.750654 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:30.750496 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:30.845127 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:30.845094 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:31.093599 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:31.093410 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:31.162503 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:31.162455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:31.598011 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:31.597848 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:31.598402 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:31.598140 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:31.598402 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:31.597870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:31.598402 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:31.598246 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:31.681503 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:31.678071 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r9jp7"] Aug 04 19:16:31.681503 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:31.678211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:31.681503 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:31.678344 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:31.684224 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:31.684110 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wk54h"] Aug 04 19:16:31.684964 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:31.684937 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t9kwv"] Aug 04 19:16:31.853184 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:31.853112 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:31.853184 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:31.853128 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:31.853364 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:31.853251 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:31.853419 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:31.853382 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:32.138651 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:32.137392 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:32.198402 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:32.198359 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:33.597747 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:33.597709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:33.598672 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:33.597715 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:33.598672 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:33.597843 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:33.598672 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:33.597864 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:33.598672 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:33.597911 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:33.598672 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:33.597977 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:34.861356 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:34.861325 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b803896-90de-4732-bee5-3c7a93aa215e" containerID="bd83c270482754fa354b7ab7273489e1fd101b6148cdc3a50e37ea5d97296906" exitCode=0 Aug 04 19:16:34.861833 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:34.861380 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vkf" event={"ID":"3b803896-90de-4732-bee5-3c7a93aa215e","Type":"ContainerDied","Data":"bd83c270482754fa354b7ab7273489e1fd101b6148cdc3a50e37ea5d97296906"} Aug 04 19:16:35.106229 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:35.106194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:35.205653 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:35.205613 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:35.574602 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:35.574531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:35.598833 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:35.598804 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:35.598982 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:35.598886 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9kwv" podUID="4e6ee10d-1bcc-42fc-911d-76886bc6e292" Aug 04 19:16:35.598982 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:35.598945 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:35.598982 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:35.598952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:35.599146 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:35.599037 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9jp7" podUID="241e4560-d288-44ae-af3a-49be302f4d6a" Aug 04 19:16:35.599146 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:35.599125 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk54h" podUID="3747b048-0cd0-42b4-994f-4df2da8e8641" Aug 04 19:16:35.674729 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:35.674691 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:35.866649 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:35.866560 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b803896-90de-4732-bee5-3c7a93aa215e" containerID="f15370cc40bd154aae1ec35c72a9b450349425b238a93887db51bac84573174b" exitCode=0 Aug 04 19:16:35.866649 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:35.866608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vkf" event={"ID":"3b803896-90de-4732-bee5-3c7a93aa215e","Type":"ContainerDied","Data":"f15370cc40bd154aae1ec35c72a9b450349425b238a93887db51bac84573174b"} Aug 04 19:16:36.394491 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.394453 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeReady" Aug 04 19:16:36.394622 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.394572 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 19:16:36.443758 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.443733 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sjsm5"] Aug 04 19:16:36.464362 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.464342 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rn9wg"] Aug 04 19:16:36.464512 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.464492 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.467029 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.467008 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 19:16:36.467178 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.467159 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svf4z\"" Aug 04 19:16:36.467367 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.467343 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 19:16:36.486368 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.486348 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sjsm5"] Aug 04 19:16:36.486442 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.486401 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rn9wg"] Aug 04 19:16:36.486442 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.486348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rn9wg" Aug 04 19:16:36.492077 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.492061 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 19:16:36.492171 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.492156 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sds9s\"" Aug 04 19:16:36.492711 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.492689 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 19:16:36.492711 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.492692 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 19:16:36.568293 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.568272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-tmp-dir\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.568391 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.568340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthmm\" (UniqueName: \"kubernetes.io/projected/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-kube-api-access-gthmm\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.568391 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.568359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn4jm\" (UniqueName: \"kubernetes.io/projected/eadc261a-9b4f-493b-a3c9-ba73bcac9dd9-kube-api-access-gn4jm\") pod \"ingress-canary-rn9wg\" (UID: \"eadc261a-9b4f-493b-a3c9-ba73bcac9dd9\") " pod="openshift-ingress-canary/ingress-canary-rn9wg" Aug 04 19:16:36.568391 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.568378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eadc261a-9b4f-493b-a3c9-ba73bcac9dd9-cert\") pod \"ingress-canary-rn9wg\" (UID: \"eadc261a-9b4f-493b-a3c9-ba73bcac9dd9\") " pod="openshift-ingress-canary/ingress-canary-rn9wg" Aug 04 19:16:36.568530 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.568430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-metrics-tls\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.568530 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.568459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-config-volume\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.609119 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.609073 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:36.669015 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.668988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-tmp-dir\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.669128 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.669051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthmm\" (UniqueName: \"kubernetes.io/projected/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-kube-api-access-gthmm\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.669128 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.669078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn4jm\" (UniqueName: \"kubernetes.io/projected/eadc261a-9b4f-493b-a3c9-ba73bcac9dd9-kube-api-access-gn4jm\") pod \"ingress-canary-rn9wg\" (UID: \"eadc261a-9b4f-493b-a3c9-ba73bcac9dd9\") " pod="openshift-ingress-canary/ingress-canary-rn9wg" Aug 04 19:16:36.669128 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.669101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eadc261a-9b4f-493b-a3c9-ba73bcac9dd9-cert\") pod \"ingress-canary-rn9wg\" (UID: \"eadc261a-9b4f-493b-a3c9-ba73bcac9dd9\") " pod="openshift-ingress-canary/ingress-canary-rn9wg" Aug 04 19:16:36.669280 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.669137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-metrics-tls\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.669280 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.669165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-config-volume\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.669402 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.669381 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-tmp-dir\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.669704 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.669686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-config-volume\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.673211 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.673190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-metrics-tls\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.673313 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.673236 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eadc261a-9b4f-493b-a3c9-ba73bcac9dd9-cert\") pod \"ingress-canary-rn9wg\" (UID: \"eadc261a-9b4f-493b-a3c9-ba73bcac9dd9\") " pod="openshift-ingress-canary/ingress-canary-rn9wg" Aug 04 19:16:36.677482 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.677442 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn4jm\" (UniqueName: \"kubernetes.io/projected/eadc261a-9b4f-493b-a3c9-ba73bcac9dd9-kube-api-access-gn4jm\") pod \"ingress-canary-rn9wg\" (UID: \"eadc261a-9b4f-493b-a3c9-ba73bcac9dd9\") " pod="openshift-ingress-canary/ingress-canary-rn9wg" Aug 04 19:16:36.677562 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.677535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthmm\" (UniqueName: \"kubernetes.io/projected/456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b-kube-api-access-gthmm\") pod \"dns-default-sjsm5\" (UID: \"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b\") " pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.774092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.774017 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:36.795148 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.795130 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rn9wg" Aug 04 19:16:36.872821 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.872784 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vkf" event={"ID":"3b803896-90de-4732-bee5-3c7a93aa215e","Type":"ContainerStarted","Data":"650be21ccc17c52b144d16dd4ea163738d9a6a477787ceb0d41eac90c0a4d9e4"} Aug 04 19:16:36.897289 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.897223 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-94vkf" podStartSLOduration=8.28150685 podStartE2EDuration="40.897201506s" totalImagesPullingTime="32.615694656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:36.895444916 +0000 UTC m=+41.801045456" watchObservedRunningTime="2026-08-04 19:16:36.897201506 +0000 UTC m=+41.802802047" Aug 04 19:16:36.969150 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.969119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rn9wg"] Aug 04 19:16:36.972074 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:16:36.972038 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadc261a_9b4f_493b_a3c9_ba73bcac9dd9.slice/crio-23ec1f5144dcca68a97059b3a2cd3fa0c49b8ca5a0053cbc3e049304e2961615 WatchSource:0}: Error finding container 23ec1f5144dcca68a97059b3a2cd3fa0c49b8ca5a0053cbc3e049304e2961615: Status 404 returned error can't find the container with id 23ec1f5144dcca68a97059b3a2cd3fa0c49b8ca5a0053cbc3e049304e2961615 Aug 04 19:16:36.980178 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.980156 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sjsm5"] Aug 04 19:16:36.985194 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:36.985169 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:37.044630 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.044569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:37.102513 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.102460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:37.598160 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.598124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:16:37.598322 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.598127 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:16:37.598322 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.598124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:16:37.602674 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.602653 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 19:16:37.602777 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.602653 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 19:16:37.603233 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.603212 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bhjx\"" Aug 04 19:16:37.603699 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.603684 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 19:16:37.603891 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.603877 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pr7h8\"" Aug 04 19:16:37.604102 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.604090 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 19:16:37.876809 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.876726 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rn9wg" event={"ID":"eadc261a-9b4f-493b-a3c9-ba73bcac9dd9","Type":"ContainerStarted","Data":"23ec1f5144dcca68a97059b3a2cd3fa0c49b8ca5a0053cbc3e049304e2961615"} Aug 04 19:16:37.878147 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:37.878114 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sjsm5" event={"ID":"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b","Type":"ContainerStarted","Data":"d1b0d39910ed5e1a492aefbcf2a9be8ca122bbfe17a87a37466a31598e0ca3ee"} Aug 04 19:16:38.504973 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.504937 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d6c9b5f4c-n79qt"] Aug 04 19:16:38.529262 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.529239 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d6c9b5f4c-n79qt"] Aug 04 19:16:38.529410 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.529332 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.532361 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.532338 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 19:16:38.532522 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.532376 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 19:16:38.532522 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.532377 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 19:16:38.532522 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.532338 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 19:16:38.533609 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.533590 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 19:16:38.533814 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.533609 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 19:16:38.533814 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.533636 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 19:16:38.533908 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.533843 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5wcwc\"" Aug 04 19:16:38.537812 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.537791 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 19:16:38.582278 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.582249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-serving-cert\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.582278 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.582276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-trusted-ca-bundle\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.582404 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.582294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-service-ca\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.582404 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.582313 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-oauth-serving-cert\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.582404 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.582355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-config\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.582540 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.582446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-oauth-config\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.582540 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.582482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8848z\" (UniqueName: \"kubernetes.io/projected/d3be89e3-d661-4b27-a5eb-7bd3df497e34-kube-api-access-8848z\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.683726 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.683697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-oauth-config\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.683823 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.683729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8848z\" (UniqueName: \"kubernetes.io/projected/d3be89e3-d661-4b27-a5eb-7bd3df497e34-kube-api-access-8848z\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.683823 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.683794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-serving-cert\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.683823 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.683811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-trusted-ca-bundle\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.683954 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.683834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-service-ca\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.683954 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.683861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-oauth-serving-cert\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.683954 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.683889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-config\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.684550 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.684526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-service-ca\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.684790 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.684768 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-trusted-ca-bundle\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.684858 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.684824 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-oauth-serving-cert\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.684919 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.684899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-config\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.686279 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.686263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-serving-cert\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.686441 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.686421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-oauth-config\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.692605 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.692581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8848z\" (UniqueName: \"kubernetes.io/projected/d3be89e3-d661-4b27-a5eb-7bd3df497e34-kube-api-access-8848z\") pod \"console-5d6c9b5f4c-n79qt\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:38.839154 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:38.839083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:39.133592 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:39.133567 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d6c9b5f4c-n79qt"] Aug 04 19:16:39.135824 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:16:39.135785 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3be89e3_d661_4b27_a5eb_7bd3df497e34.slice/crio-582a532030e24c43a9ae18294c0b66d35aca2a4270bb5c3052f30cdf775aed93 WatchSource:0}: Error finding container 582a532030e24c43a9ae18294c0b66d35aca2a4270bb5c3052f30cdf775aed93: Status 404 returned error can't find the container with id 582a532030e24c43a9ae18294c0b66d35aca2a4270bb5c3052f30cdf775aed93 Aug 04 19:16:39.884074 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:39.884039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6c9b5f4c-n79qt" event={"ID":"d3be89e3-d661-4b27-a5eb-7bd3df497e34","Type":"ContainerStarted","Data":"582a532030e24c43a9ae18294c0b66d35aca2a4270bb5c3052f30cdf775aed93"} Aug 04 19:16:39.886719 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:39.886694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sjsm5" event={"ID":"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b","Type":"ContainerStarted","Data":"77129aaf3d6c3135bcac2c7c9724847ac3dc9230518a2bc7ad2ffd45e0948508"} Aug 04 19:16:39.886833 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:39.886742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sjsm5" event={"ID":"456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b","Type":"ContainerStarted","Data":"7f929bf8e74546ad37651fca21e1b054b27f5863c36d3f401a9cf18a8a2a66f5"} Aug 04 19:16:39.886833 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:39.886823 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:39.895136 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:39.895106 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:39.904564 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:39.904514 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sjsm5" podStartSLOduration=1.899902189 podStartE2EDuration="3.904497148s" totalImagesPullingTime="2.004594959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:39.904082612 +0000 UTC m=+44.809683152" watchObservedRunningTime="2026-08-04 19:16:39.904497148 +0000 UTC m=+44.810097690" Aug 04 19:16:39.996069 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:39.995888 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:40.115692 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:40.115654 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:40.170210 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:40.170118 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:40.958479 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:40.958431 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:41.050308 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.050276 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:41.136698 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.136669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:41.156592 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.156567 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6"] Aug 04 19:16:41.176146 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.176108 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6"] Aug 04 19:16:41.176558 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.176225 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" Aug 04 19:16:41.179016 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.178995 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Aug 04 19:16:41.179116 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.179019 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-khzm8\"" Aug 04 19:16:41.206534 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.206513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a72f3fab-ef8e-44e0-987a-0995ab9b03de-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-gdfl6\" (UID: \"a72f3fab-ef8e-44e0-987a-0995ab9b03de\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" Aug 04 19:16:41.265212 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.265148 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mk4d7"] Aug 04 19:16:41.281261 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.281230 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mk4d7"] Aug 04 19:16:41.281382 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.281339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.284171 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.284152 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 19:16:41.284353 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.284332 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:41.284401 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.284363 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 19:16:41.284607 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.284587 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 19:16:41.284733 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.284682 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m6xxp\"" Aug 04 19:16:41.284831 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.284814 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 19:16:41.307452 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.307433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a72f3fab-ef8e-44e0-987a-0995ab9b03de-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-gdfl6\" (UID: \"a72f3fab-ef8e-44e0-987a-0995ab9b03de\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" Aug 04 19:16:41.307568 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.307480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.307568 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.307516 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.307568 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.307545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0784792e-39c0-4fa0-8f74-e3a016877e43-crio-socket\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.307729 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:41.307583 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Aug 04 19:16:41.307729 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:41.307635 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72f3fab-ef8e-44e0-987a-0995ab9b03de-tls-certificates podName:a72f3fab-ef8e-44e0-987a-0995ab9b03de nodeName:}" failed. No retries permitted until 2026-08-04 19:16:41.807618478 +0000 UTC m=+46.713219000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a72f3fab-ef8e-44e0-987a-0995ab9b03de-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-gdfl6" (UID: "a72f3fab-ef8e-44e0-987a-0995ab9b03de") : secret "prometheus-operator-admission-webhook-tls" not found Aug 04 19:16:41.307729 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.307631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqzc\" (UniqueName: \"kubernetes.io/projected/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-api-access-9wqzc\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.307729 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.307692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0784792e-39c0-4fa0-8f74-e3a016877e43-data-volume\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.361649 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.361613 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:41.408902 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.408871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.409028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.408917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.409028 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:41.408994 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:41.409115 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:41.409030 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:41.409115 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.409042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0784792e-39c0-4fa0-8f74-e3a016877e43-crio-socket\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.409115 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:41.409067 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm podName:0784792e-39c0-4fa0-8f74-e3a016877e43 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:41.909048247 +0000 UTC m=+46.814648764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mk4d7" (UID: "0784792e-39c0-4fa0-8f74-e3a016877e43") : configmap "kube-rbac-proxy" not found Aug 04 19:16:41.409237 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:41.409133 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls podName:0784792e-39c0-4fa0-8f74-e3a016877e43 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:41.909101269 +0000 UTC m=+46.814701805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk4d7" (UID: "0784792e-39c0-4fa0-8f74-e3a016877e43") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:41.409237 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.409200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqzc\" (UniqueName: \"kubernetes.io/projected/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-api-access-9wqzc\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.409237 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.409212 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0784792e-39c0-4fa0-8f74-e3a016877e43-crio-socket\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.409354 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.409254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0784792e-39c0-4fa0-8f74-e3a016877e43-data-volume\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.409675 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.409649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0784792e-39c0-4fa0-8f74-e3a016877e43-data-volume\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.418375 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.418349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqzc\" (UniqueName: \"kubernetes.io/projected/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-api-access-9wqzc\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.462735 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.462707 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:41.812486 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.812430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a72f3fab-ef8e-44e0-987a-0995ab9b03de-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-gdfl6\" (UID: \"a72f3fab-ef8e-44e0-987a-0995ab9b03de\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" Aug 04 19:16:41.815360 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.815333 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a72f3fab-ef8e-44e0-987a-0995ab9b03de-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-gdfl6\" (UID: \"a72f3fab-ef8e-44e0-987a-0995ab9b03de\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" Aug 04 19:16:41.913499 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.912970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.913499 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:41.913088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:41.913499 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:41.913179 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:41.913499 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:41.913244 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm podName:0784792e-39c0-4fa0-8f74-e3a016877e43 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:42.913226147 +0000 UTC m=+47.818826668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mk4d7" (UID: "0784792e-39c0-4fa0-8f74-e3a016877e43") : configmap "kube-rbac-proxy" not found Aug 04 19:16:41.913842 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:41.913626 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:41.913842 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:41.913681 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls podName:0784792e-39c0-4fa0-8f74-e3a016877e43 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:42.913662852 +0000 UTC m=+47.819263376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk4d7" (UID: "0784792e-39c0-4fa0-8f74-e3a016877e43") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:42.087097 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.087012 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" Aug 04 19:16:42.196874 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.196821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:42.268632 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.268599 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:42.400051 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.400025 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6"] Aug 04 19:16:42.402757 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:16:42.402737 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72f3fab_ef8e_44e0_987a_0995ab9b03de.slice/crio-51eab8fc27c608db3d58c61d806e7a22fa66c88500974c07db612cbfaf03f927 WatchSource:0}: Error finding container 51eab8fc27c608db3d58c61d806e7a22fa66c88500974c07db612cbfaf03f927: Status 404 returned error can't find the container with id 51eab8fc27c608db3d58c61d806e7a22fa66c88500974c07db612cbfaf03f927 Aug 04 19:16:42.895859 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.895780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rn9wg" event={"ID":"eadc261a-9b4f-493b-a3c9-ba73bcac9dd9","Type":"ContainerStarted","Data":"d954cfd1ba5b7db675d027bf0ad1590e118773af317e17b4362e9b2fb20ff2b5"} Aug 04 19:16:42.897296 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.897273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6c9b5f4c-n79qt" event={"ID":"d3be89e3-d661-4b27-a5eb-7bd3df497e34","Type":"ContainerStarted","Data":"952b498c8c700ab9a739dff81c61328a1acb01463506434872af7263f041a600"} Aug 04 19:16:42.898357 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.898340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" event={"ID":"a72f3fab-ef8e-44e0-987a-0995ab9b03de","Type":"ContainerStarted","Data":"51eab8fc27c608db3d58c61d806e7a22fa66c88500974c07db612cbfaf03f927"} Aug 04 19:16:42.912444 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.912387 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rn9wg" podStartSLOduration=1.60352233 podStartE2EDuration="6.91237053s" totalImagesPullingTime="5.3088482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:42.910505119 +0000 UTC m=+47.816105670" watchObservedRunningTime="2026-08-04 19:16:42.91237053 +0000 UTC m=+47.817971072" Aug 04 19:16:42.923256 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.923156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:42.923256 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.923227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:42.923443 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:42.923358 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:42.923443 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:42.923437 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm podName:0784792e-39c0-4fa0-8f74-e3a016877e43 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:44.923417604 +0000 UTC m=+49.829018141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mk4d7" (UID: "0784792e-39c0-4fa0-8f74-e3a016877e43") : configmap "kube-rbac-proxy" not found Aug 04 19:16:42.923860 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:42.923365 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:42.923860 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:42.923698 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls podName:0784792e-39c0-4fa0-8f74-e3a016877e43 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:44.923677449 +0000 UTC m=+49.829277980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk4d7" (UID: "0784792e-39c0-4fa0-8f74-e3a016877e43") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:42.929285 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:42.929137 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d6c9b5f4c-n79qt" podStartSLOduration=1.4353252140000001 podStartE2EDuration="4.929119237s" totalImagesPullingTime="3.493794023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:42.9275876 +0000 UTC m=+47.833188140" watchObservedRunningTime="2026-08-04 19:16:42.929119237 +0000 UTC m=+47.834719777" Aug 04 19:16:43.852578 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:43.852557 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d6c9b5f4c-n79qt_d3be89e3-d661-4b27-a5eb-7bd3df497e34/console/0.log" Aug 04 19:16:43.905508 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:43.905454 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" event={"ID":"a72f3fab-ef8e-44e0-987a-0995ab9b03de","Type":"ContainerStarted","Data":"d455c8e80ffdbad952eb44525c7f0cfb6dfb926c4a1284b4405f2789765fd47f"} Aug 04 19:16:43.921591 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:43.921538 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" podStartSLOduration=1.542450828 podStartE2EDuration="2.921517974s" totalImagesPullingTime="1.379067146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:43.920447802 +0000 UTC m=+48.826048341" watchObservedRunningTime="2026-08-04 19:16:43.921517974 +0000 UTC m=+48.827118518" Aug 04 19:16:44.909106 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:44.909072 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" Aug 04 19:16:44.913610 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:44.913590 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-gdfl6" Aug 04 19:16:44.936376 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:44.936355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:44.936449 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:44.936386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:44.936515 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:44.936481 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:44.936515 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:44.936501 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:44.936582 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:44.936545 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm podName:0784792e-39c0-4fa0-8f74-e3a016877e43 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:48.936527092 +0000 UTC m=+53.842127628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mk4d7" (UID: "0784792e-39c0-4fa0-8f74-e3a016877e43") : configmap "kube-rbac-proxy" not found Aug 04 19:16:44.936582 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:44.936566 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls podName:0784792e-39c0-4fa0-8f74-e3a016877e43 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:48.936555577 +0000 UTC m=+53.842156094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk4d7" (UID: "0784792e-39c0-4fa0-8f74-e3a016877e43") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:45.052893 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.052864 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sjsm5_456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b/dns/0.log" Aug 04 19:16:45.191045 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.190996 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:45.250919 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.250892 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-5xjhq"] Aug 04 19:16:45.256323 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.256300 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sjsm5_456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b/kube-rbac-proxy/0.log" Aug 04 19:16:45.264488 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.264453 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.266805 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.266786 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-5xjhq"] Aug 04 19:16:45.267844 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.267821 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 19:16:45.267944 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.267918 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Aug 04 19:16:45.269774 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.269756 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Aug 04 19:16:45.269774 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.269765 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qzvt6\"" Aug 04 19:16:45.269968 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.269956 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 19:16:45.270024 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.269970 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 19:16:45.288613 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.288589 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:45.338962 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.338939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/51521b28-c77b-4315-a048-d627eafc3751-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.339052 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.338973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51521b28-c77b-4315-a048-d627eafc3751-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.339052 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.339028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51521b28-c77b-4315-a048-d627eafc3751-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.339132 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.339053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svkw\" (UniqueName: \"kubernetes.io/projected/51521b28-c77b-4315-a048-d627eafc3751-kube-api-access-8svkw\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.440078 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.440040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51521b28-c77b-4315-a048-d627eafc3751-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.440078 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.440078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svkw\" (UniqueName: \"kubernetes.io/projected/51521b28-c77b-4315-a048-d627eafc3751-kube-api-access-8svkw\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.440261 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.440141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/51521b28-c77b-4315-a048-d627eafc3751-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.440261 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.440167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51521b28-c77b-4315-a048-d627eafc3751-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.440318 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:45.440274 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Aug 04 19:16:45.440384 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:45.440370 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51521b28-c77b-4315-a048-d627eafc3751-prometheus-operator-tls podName:51521b28-c77b-4315-a048-d627eafc3751 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:45.940349231 +0000 UTC m=+50.845949767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/51521b28-c77b-4315-a048-d627eafc3751-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-5xjhq" (UID: "51521b28-c77b-4315-a048-d627eafc3751") : secret "prometheus-operator-tls" not found Aug 04 19:16:45.440901 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.440832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51521b28-c77b-4315-a048-d627eafc3751-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.442662 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.442614 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51521b28-c77b-4315-a048-d627eafc3751-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.452883 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.452858 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svkw\" (UniqueName: \"kubernetes.io/projected/51521b28-c77b-4315-a048-d627eafc3751-kube-api-access-8svkw\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.658943 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.658916 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gk8cr_c261189f-7399-4ff4-a82c-06f9f682305d/dns-node-resolver/0.log" Aug 04 19:16:45.685010 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.684972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:45.943857 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.943831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/51521b28-c77b-4315-a048-d627eafc3751-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:45.946257 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:45.946237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/51521b28-c77b-4315-a048-d627eafc3751-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5xjhq\" (UID: \"51521b28-c77b-4315-a048-d627eafc3751\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:46.175276 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:46.175235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" Aug 04 19:16:46.317526 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:46.317395 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-5xjhq"] Aug 04 19:16:46.319876 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:16:46.319851 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51521b28_c77b_4315_a048_d627eafc3751.slice/crio-55545e5d69638b6b88e0940af59d6c023dfa164349c0d2435743fa53e1130851 WatchSource:0}: Error finding container 55545e5d69638b6b88e0940af59d6c023dfa164349c0d2435743fa53e1130851: Status 404 returned error can't find the container with id 55545e5d69638b6b88e0940af59d6c023dfa164349c0d2435743fa53e1130851 Aug 04 19:16:46.652555 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:46.652481 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvzq8_14c3bacb-2797-4096-ace0-d6e5b61e26be/node-ca/0.log" Aug 04 19:16:46.916796 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:46.916726 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" event={"ID":"51521b28-c77b-4315-a048-d627eafc3751","Type":"ContainerStarted","Data":"55545e5d69638b6b88e0940af59d6c023dfa164349c0d2435743fa53e1130851"} Aug 04 19:16:47.652377 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:47.652353 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rn9wg_eadc261a-9b4f-493b-a3c9-ba73bcac9dd9/serve-healthcheck-canary/0.log" Aug 04 19:16:47.980929 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:47.980890 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:48.049572 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:48.049540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:48.840073 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:48.840033 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:48.840457 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:48.840100 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:48.845103 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:48.845077 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:48.928981 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:48.928959 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:16:48.968341 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:48.968315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:48.968519 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:48.968373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:48.968519 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:48.968482 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:48.968643 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:48.968561 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm podName:0784792e-39c0-4fa0-8f74-e3a016877e43 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:56.968539542 +0000 UTC m=+61.874140077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mk4d7" (UID: "0784792e-39c0-4fa0-8f74-e3a016877e43") : configmap "kube-rbac-proxy" not found Aug 04 19:16:48.968643 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:48.968580 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:48.968643 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:48.968634 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls podName:0784792e-39c0-4fa0-8f74-e3a016877e43 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:56.968621794 +0000 UTC m=+61.874222311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk4d7" (UID: "0784792e-39c0-4fa0-8f74-e3a016877e43") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:49.100362 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:49.100300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:49.200845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:49.200777 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:49.302647 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:49.302617 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:49.359457 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:49.359374 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:49.871684 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:49.871652 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:49.891367 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:49.891346 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sjsm5" Aug 04 19:16:49.929589 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:49.929555 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" event={"ID":"51521b28-c77b-4315-a048-d627eafc3751","Type":"ContainerStarted","Data":"dfcc9580cef65848b1df0515737770512333240721ae132b7e6a371f42a73c80"} Aug 04 19:16:49.929710 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:49.929621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" event={"ID":"51521b28-c77b-4315-a048-d627eafc3751","Type":"ContainerStarted","Data":"d235f8fd91c018e41ac76be76ccd58f6bcc9a496d42c3836860a51ed0644dbfc"} Aug 04 19:16:49.946408 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:49.946339 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-5xjhq" podStartSLOduration=2.404049796 podStartE2EDuration="4.9463236s" totalImagesPullingTime="2.542273804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:49.945592434 +0000 UTC m=+54.851192972" watchObservedRunningTime="2026-08-04 19:16:49.9463236 +0000 UTC m=+54.851924121" Aug 04 19:16:49.950362 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:49.950334 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:51.417738 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.417700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:51.648161 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.648121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:51.654087 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.654063 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm"] Aug 04 19:16:51.656063 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.656045 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.659126 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.659103 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-42sgw\"" Aug 04 19:16:51.659224 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.659147 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 04 19:16:51.659224 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.659199 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 04 19:16:51.669539 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.669437 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm"] Aug 04 19:16:51.710681 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.710658 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nlpxg"] Aug 04 19:16:51.712529 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.712515 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.715346 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.715330 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 19:16:51.715619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.715602 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 19:16:51.715880 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.715863 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mrvps\"" Aug 04 19:16:51.716223 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.716208 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 19:16:51.793231 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.793362 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/759c21dc-d5ce-48ed-b6be-2acbee04895a-sys\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.793362 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-tls\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.793362 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/759c21dc-d5ce-48ed-b6be-2acbee04895a-root\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.793362 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.793614 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759c21dc-d5ce-48ed-b6be-2acbee04895a-metrics-client-ca\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.793614 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-wtmp\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.793614 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.793614 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793577 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ssg\" (UniqueName: \"kubernetes.io/projected/759c21dc-d5ce-48ed-b6be-2acbee04895a-kube-api-access-d7ssg\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.793759 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.793759 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-textfile\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.793759 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.793759 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.793698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5ppk\" (UniqueName: \"kubernetes.io/projected/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-kube-api-access-d5ppk\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.894122 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759c21dc-d5ce-48ed-b6be-2acbee04895a-metrics-client-ca\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894247 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-wtmp\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894247 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894247 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ssg\" (UniqueName: \"kubernetes.io/projected/759c21dc-d5ce-48ed-b6be-2acbee04895a-kube-api-access-d7ssg\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894247 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894449 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-textfile\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894449 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894291 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-wtmp\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894449 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.894449 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:51.894358 2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Aug 04 19:16:51.894449 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:51.894412 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-openshift-state-metrics-tls podName:6edbc642-c8d7-497d-b2a9-fe6c03d34c09 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:52.394395129 +0000 UTC m=+57.299995652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-7nqvm" (UID: "6edbc642-c8d7-497d-b2a9-fe6c03d34c09") : secret "openshift-state-metrics-tls" not found Aug 04 19:16:51.894737 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5ppk\" (UniqueName: \"kubernetes.io/projected/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-kube-api-access-d5ppk\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.894737 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.894737 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/759c21dc-d5ce-48ed-b6be-2acbee04895a-sys\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894737 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-tls\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894737 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/759c21dc-d5ce-48ed-b6be-2acbee04895a-root\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894737 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.894737 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894677 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-textfile\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894737 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/759c21dc-d5ce-48ed-b6be-2acbee04895a-sys\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.894737 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:51.894707 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 19:16:51.895136 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/759c21dc-d5ce-48ed-b6be-2acbee04895a-root\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.895136 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:16:51.894815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-tls podName:759c21dc-d5ce-48ed-b6be-2acbee04895a nodeName:}" failed. No retries permitted until 2026-08-04 19:16:52.394797892 +0000 UTC m=+57.300398411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-tls") pod "node-exporter-nlpxg" (UID: "759c21dc-d5ce-48ed-b6be-2acbee04895a") : secret "node-exporter-tls" not found Aug 04 19:16:51.895136 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894862 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759c21dc-d5ce-48ed-b6be-2acbee04895a-metrics-client-ca\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.895136 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.894880 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.895305 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.895236 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.896965 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.896945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:51.897044 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.896948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.903714 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.903684 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ssg\" (UniqueName: \"kubernetes.io/projected/759c21dc-d5ce-48ed-b6be-2acbee04895a-kube-api-access-d7ssg\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:51.904586 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:51.904568 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5ppk\" (UniqueName: \"kubernetes.io/projected/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-kube-api-access-d5ppk\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:52.399485 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.399430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:52.399686 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.399537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-tls\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:52.401894 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.401863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/759c21dc-d5ce-48ed-b6be-2acbee04895a-node-exporter-tls\") pod \"node-exporter-nlpxg\" (UID: \"759c21dc-d5ce-48ed-b6be-2acbee04895a\") " pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:52.402056 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.402036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edbc642-c8d7-497d-b2a9-fe6c03d34c09-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-7nqvm\" (UID: \"6edbc642-c8d7-497d-b2a9-fe6c03d34c09\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:52.460329 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.460300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:52.562239 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.562205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:52.565046 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.565028 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" Aug 04 19:16:52.621030 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.621001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nlpxg" Aug 04 19:16:52.634240 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:16:52.634210 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759c21dc_d5ce_48ed_b6be_2acbee04895a.slice/crio-f62e0ddf1c4733f61e568459e9501b18dfc04bc4255fb04e795ad1954390b78b WatchSource:0}: Error finding container f62e0ddf1c4733f61e568459e9501b18dfc04bc4255fb04e795ad1954390b78b: Status 404 returned error can't find the container with id f62e0ddf1c4733f61e568459e9501b18dfc04bc4255fb04e795ad1954390b78b Aug 04 19:16:52.661185 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.661063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:52.708015 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.707988 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm"] Aug 04 19:16:52.761183 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.761156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:52.832198 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.832173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:52.939897 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.939862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" event={"ID":"6edbc642-c8d7-497d-b2a9-fe6c03d34c09","Type":"ContainerStarted","Data":"b10a453cfa13056a5b9b5c71a2ceb641489ff024ada1cfd17acd1e97d14423b7"} Aug 04 19:16:52.940038 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.939954 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" event={"ID":"6edbc642-c8d7-497d-b2a9-fe6c03d34c09","Type":"ContainerStarted","Data":"9ac3ac09a6174ec8eb603a33b0b5c2870e1c32902853c5adf0f7ccc76a89a3d4"} Aug 04 19:16:52.940038 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.939974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" event={"ID":"6edbc642-c8d7-497d-b2a9-fe6c03d34c09","Type":"ContainerStarted","Data":"5bf93aa988bc38b844b04424f7ee23896933d45cc06f5ce975ef7e1ab77c7b75"} Aug 04 19:16:52.940978 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:52.940957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlpxg" event={"ID":"759c21dc-d5ce-48ed-b6be-2acbee04895a","Type":"ContainerStarted","Data":"f62e0ddf1c4733f61e568459e9501b18dfc04bc4255fb04e795ad1954390b78b"} Aug 04 19:16:53.163982 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:53.163949 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:53.927765 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:53.927726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:53.946053 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:53.946031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" event={"ID":"6edbc642-c8d7-497d-b2a9-fe6c03d34c09","Type":"ContainerStarted","Data":"560687f535c14ca20aa3a7d5ea23dc4ea526660a4fa1009135f381f1975c874c"} Aug 04 19:16:53.947446 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:53.947422 2580 generic.go:356] "Generic (PLEG): container finished" podID="759c21dc-d5ce-48ed-b6be-2acbee04895a" containerID="7c6118eb658a8e255f70cb03132b8e76645b192927bcfd5322f7bef7eb967d0a" exitCode=0 Aug 04 19:16:53.947554 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:53.947483 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlpxg" event={"ID":"759c21dc-d5ce-48ed-b6be-2acbee04895a","Type":"ContainerDied","Data":"7c6118eb658a8e255f70cb03132b8e76645b192927bcfd5322f7bef7eb967d0a"} Aug 04 19:16:53.962199 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:53.962161 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7nqvm" podStartSLOduration=1.991131514 podStartE2EDuration="2.962147257s" totalImagesPullingTime="971.015743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:53.961642786 +0000 UTC m=+58.867243325" watchObservedRunningTime="2026-08-04 19:16:53.962147257 +0000 UTC m=+58.867747796" Aug 04 19:16:54.028195 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.028168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:54.765871 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.765836 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5b49478744-q9h8w"] Aug 04 19:16:54.768245 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.768230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.771020 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.770998 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 04 19:16:54.771154 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.771026 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 04 19:16:54.771154 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.771085 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 04 19:16:54.771355 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.771339 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a3r8n69cp2pic\"" Aug 04 19:16:54.771420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.771390 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 04 19:16:54.771420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.771397 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4sdvx\"" Aug 04 19:16:54.772095 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.772079 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 04 19:16:54.780812 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.780789 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b49478744-q9h8w"] Aug 04 19:16:54.822234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.822210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-grpc-tls\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.822329 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.822247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-tls\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.822329 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.822273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.822329 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.822294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.822329 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.822318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-metrics-client-ca\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.822486 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.822392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.822486 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.822418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp56r\" (UniqueName: \"kubernetes.io/projected/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-kube-api-access-vp56r\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.822486 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.822451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.923064 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.923043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.923172 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.923071 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp56r\" (UniqueName: \"kubernetes.io/projected/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-kube-api-access-vp56r\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.923172 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.923103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.923172 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.923159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-grpc-tls\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.923279 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.923210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-tls\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.923279 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.923250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.923367 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.923309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.923367 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.923338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-metrics-client-ca\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.924305 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.924281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-metrics-client-ca\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.925963 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.925931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-tls\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.926066 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.926046 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.926261 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.926237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.926335 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.926290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.926393 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.926337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.926456 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.926434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-secret-grpc-tls\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.935995 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.935971 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp56r\" (UniqueName: \"kubernetes.io/projected/cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a-kube-api-access-vp56r\") pod \"thanos-querier-5b49478744-q9h8w\" (UID: \"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a\") " pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:54.952381 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.952354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlpxg" event={"ID":"759c21dc-d5ce-48ed-b6be-2acbee04895a","Type":"ContainerStarted","Data":"b2a01b7195168d1d5a94c5ff00ced945e0178b6cc45814517f39e295f4d4ae2f"} Aug 04 19:16:54.952502 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.952422 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlpxg" event={"ID":"759c21dc-d5ce-48ed-b6be-2acbee04895a","Type":"ContainerStarted","Data":"7bb9b9d3543146c18b625d0570e0abd0a025baa962de8d9714b4a34111ae6c36"} Aug 04 19:16:54.971709 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:54.971669 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nlpxg" podStartSLOduration=2.748213416 podStartE2EDuration="3.971653541s" totalImagesPullingTime="1.150461932s" totalInitContainerRuntime="72.978193ms" isStatefulPod=true podCreationTimestamp="2026-08-04 19:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:54.970320573 +0000 UTC m=+59.875921111" watchObservedRunningTime="2026-08-04 19:16:54.971653541 +0000 UTC m=+59.877254081" Aug 04 19:16:55.077233 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:55.077177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:55.192832 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:55.192801 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:55.204707 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:55.204686 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b49478744-q9h8w"] Aug 04 19:16:55.207354 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:16:55.207332 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdda85ef_d2f4_4fc2_b6ba_d4b1d00cdc4a.slice/crio-35628df091116fde78dcc740c0b6fee3e133502e6c62c40a964f985d2fac2835 WatchSource:0}: Error finding container 35628df091116fde78dcc740c0b6fee3e133502e6c62c40a964f985d2fac2835: Status 404 returned error can't find the container with id 35628df091116fde78dcc740c0b6fee3e133502e6c62c40a964f985d2fac2835 Aug 04 19:16:55.292970 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:55.292946 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:55.848950 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:55.848918 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j26l2" Aug 04 19:16:55.956185 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:55.956148 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" event={"ID":"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a","Type":"ContainerStarted","Data":"35628df091116fde78dcc740c0b6fee3e133502e6c62c40a964f985d2fac2835"} Aug 04 19:16:55.972647 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:55.972596 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:56.061503 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.061459 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:56.430125 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.430099 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj"] Aug 04 19:16:56.432354 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.432327 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" Aug 04 19:16:56.433454 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.433426 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:56.435335 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.435314 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6wp58\"" Aug 04 19:16:56.435596 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.435581 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 04 19:16:56.450353 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.450315 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj"] Aug 04 19:16:56.520294 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.520264 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:56.536158 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.536133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8k6k\" (UniqueName: \"kubernetes.io/projected/b903dca5-597c-495b-afc5-8f7b573f9ecb-kube-api-access-h8k6k\") pod \"monitoring-plugin-6dffbbd67-69xrj\" (UID: \"b903dca5-597c-495b-afc5-8f7b573f9ecb\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" Aug 04 19:16:56.536306 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.536238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b903dca5-597c-495b-afc5-8f7b573f9ecb-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-69xrj\" (UID: \"b903dca5-597c-495b-afc5-8f7b573f9ecb\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" Aug 04 19:16:56.617361 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.617318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:56.637353 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.637325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8k6k\" (UniqueName: \"kubernetes.io/projected/b903dca5-597c-495b-afc5-8f7b573f9ecb-kube-api-access-h8k6k\") pod \"monitoring-plugin-6dffbbd67-69xrj\" (UID: \"b903dca5-597c-495b-afc5-8f7b573f9ecb\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" Aug 04 19:16:56.637517 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.637413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b903dca5-597c-495b-afc5-8f7b573f9ecb-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-69xrj\" (UID: \"b903dca5-597c-495b-afc5-8f7b573f9ecb\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" Aug 04 19:16:56.640058 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.640033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b903dca5-597c-495b-afc5-8f7b573f9ecb-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-69xrj\" (UID: \"b903dca5-597c-495b-afc5-8f7b573f9ecb\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" Aug 04 19:16:56.647107 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.647087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8k6k\" (UniqueName: \"kubernetes.io/projected/b903dca5-597c-495b-afc5-8f7b573f9ecb-kube-api-access-h8k6k\") pod \"monitoring-plugin-6dffbbd67-69xrj\" (UID: \"b903dca5-597c-495b-afc5-8f7b573f9ecb\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" Aug 04 19:16:56.718680 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.718636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:56.743572 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.743535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" Aug 04 19:16:56.829266 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.829231 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:56.874773 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.874672 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj"] Aug 04 19:16:56.877133 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:16:56.877103 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb903dca5_597c_495b_afc5_8f7b573f9ecb.slice/crio-12330b21f37cfe35556ec1eb56b0e73735255b8532f070611b707377706f4e0b WatchSource:0}: Error finding container 12330b21f37cfe35556ec1eb56b0e73735255b8532f070611b707377706f4e0b: Status 404 returned error can't find the container with id 12330b21f37cfe35556ec1eb56b0e73735255b8532f070611b707377706f4e0b Aug 04 19:16:56.961744 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:56.961712 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" event={"ID":"b903dca5-597c-495b-afc5-8f7b573f9ecb","Type":"ContainerStarted","Data":"12330b21f37cfe35556ec1eb56b0e73735255b8532f070611b707377706f4e0b"} Aug 04 19:16:57.040781 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.040696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:57.040781 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.040748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:57.041410 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.041382 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0784792e-39c0-4fa0-8f74-e3a016877e43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:57.043372 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.043349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0784792e-39c0-4fa0-8f74-e3a016877e43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk4d7\" (UID: \"0784792e-39c0-4fa0-8f74-e3a016877e43\") " pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:57.195947 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.195909 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m6xxp\"" Aug 04 19:16:57.204021 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.204000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mk4d7" Aug 04 19:16:57.522150 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.522128 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mk4d7"] Aug 04 19:16:57.526514 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:16:57.526490 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0784792e_39c0_4fa0_8f74_e3a016877e43.slice/crio-fd5d04a9e92fc02c6795dc9f6f7762d83874824c95a08bd9afbcdc26e15eafab WatchSource:0}: Error finding container fd5d04a9e92fc02c6795dc9f6f7762d83874824c95a08bd9afbcdc26e15eafab: Status 404 returned error can't find the container with id fd5d04a9e92fc02c6795dc9f6f7762d83874824c95a08bd9afbcdc26e15eafab Aug 04 19:16:57.955602 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.955566 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:57.961327 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.960051 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 19:16:57.964547 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.963216 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:57.966985 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.966768 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 04 19:16:57.966985 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.966777 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pfh9d\"" Aug 04 19:16:57.966985 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.966781 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 04 19:16:57.966985 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.966808 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 04 19:16:57.967252 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.967091 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 04 19:16:57.967488 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.967402 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 04 19:16:57.967700 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.967681 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 04 19:16:57.967778 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.967708 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 04 19:16:57.968152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.967926 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 04 19:16:57.968152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.967980 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 04 19:16:57.968152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.967987 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 04 19:16:57.968152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.968054 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 04 19:16:57.968152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.967927 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-rc8n7opkac5j\"" Aug 04 19:16:57.968152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.968109 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 04 19:16:57.968848 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.968663 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 19:16:57.968848 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.968669 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 04 19:16:57.978391 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.977908 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mk4d7" event={"ID":"0784792e-39c0-4fa0-8f74-e3a016877e43","Type":"ContainerStarted","Data":"c0c1fd09b1b66a6d1a063bbad529b03cb9c9572714177e04d8c1060c8ba06d4f"} Aug 04 19:16:57.978522 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.978487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mk4d7" event={"ID":"0784792e-39c0-4fa0-8f74-e3a016877e43","Type":"ContainerStarted","Data":"fd5d04a9e92fc02c6795dc9f6f7762d83874824c95a08bd9afbcdc26e15eafab"} Aug 04 19:16:57.978898 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.978879 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 04 19:16:57.982213 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.982180 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 19:16:57.984798 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.984777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" event={"ID":"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a","Type":"ContainerStarted","Data":"f196d9f1ed7fde7788fa4d6afc4e7d7c940615dd65e35317ddda7263bcc18ea0"} Aug 04 19:16:57.984885 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.984835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" event={"ID":"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a","Type":"ContainerStarted","Data":"1a7202a61abce3e87113296d687d0a8ec0946d8f347cf576926e6adb595cae37"} Aug 04 19:16:57.984885 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:57.984861 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" event={"ID":"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a","Type":"ContainerStarted","Data":"258ed61a0af59d446ac4195a6da393ddb6dbf315b9f0c28b230156c789fc754a"} Aug 04 19:16:58.048558 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.048498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.048713 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.048590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.048713 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.048622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw2rk\" (UniqueName: \"kubernetes.io/projected/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-kube-api-access-lw2rk\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.048805 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.048736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.048805 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.048775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.048905 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.048832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.048905 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.048873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-config\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.048905 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.048899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.049008 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.048956 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.049114 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.049022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.049301 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.049257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.049496 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.049442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.049691 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.049610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.049938 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.049806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.050088 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.050051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.050293 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.050239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.050491 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.050404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.050657 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.050588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.050800 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.050620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.050984 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.050930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.154548 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.154716 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.154716 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.154716 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-config\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.154716 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.154716 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.154716 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.154991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.155012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155531 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.155043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155531 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.155087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155531 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.155113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw2rk\" (UniqueName: \"kubernetes.io/projected/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-kube-api-access-lw2rk\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155531 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.155329 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.155941 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.155914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.156877 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.156847 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.156877 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.156869 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.157229 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.157207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.157715 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.157694 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.158967 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.158913 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.160166 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.160142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.160248 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.160205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.160663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.160639 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.161241 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.161198 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-config\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.162402 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.162353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.162948 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.162920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.163547 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.163525 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.164386 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.164330 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.164984 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.164934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.165629 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.165592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.165629 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.165615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.167728 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.167588 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.170947 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.170901 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw2rk\" (UniqueName: \"kubernetes.io/projected/a3608a9f-2b8b-4896-a64d-8b3eb9d38b04-kube-api-access-lw2rk\") pod \"prometheus-k8s-0\" (UID: \"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.279931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.279850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:16:58.804754 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.804658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 19:16:58.805992 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:16:58.805969 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3608a9f_2b8b_4896_a64d_8b3eb9d38b04.slice/crio-ca641dcbcb7c6fa195d3fab032827edce31d5210c5bac7ea388d188482720554 WatchSource:0}: Error finding container ca641dcbcb7c6fa195d3fab032827edce31d5210c5bac7ea388d188482720554: Status 404 returned error can't find the container with id ca641dcbcb7c6fa195d3fab032827edce31d5210c5bac7ea388d188482720554 Aug 04 19:16:58.991354 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.991323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" event={"ID":"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a","Type":"ContainerStarted","Data":"98eeab9a3a157d77f1b96e3bb5604710e5bb53f0f53e9f6bfe52ff9f71fc9906"} Aug 04 19:16:58.991834 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.991394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" event={"ID":"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a","Type":"ContainerStarted","Data":"55bde9e3668eb05e96a66ffd5fa8b0b8af8863cbb6fb7a90f2020246ff6e1ae2"} Aug 04 19:16:58.991834 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.991438 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" event={"ID":"cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a","Type":"ContainerStarted","Data":"13be5e472a7801c86dc0331ba1c9254c1a6df96e9e4768cda7ee14db740ccf54"} Aug 04 19:16:58.991834 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.991509 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:16:58.992681 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.992661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04","Type":"ContainerStarted","Data":"ca641dcbcb7c6fa195d3fab032827edce31d5210c5bac7ea388d188482720554"} Aug 04 19:16:58.993916 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.993893 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" event={"ID":"b903dca5-597c-495b-afc5-8f7b573f9ecb","Type":"ContainerStarted","Data":"cdb042b5beeed2dbbdaf84914ec13051585ef49116c2ca5b1eb8b94c9c01da43"} Aug 04 19:16:58.994107 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.994093 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" Aug 04 19:16:58.998753 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:58.998733 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" Aug 04 19:16:59.013166 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:59.013128 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" podStartSLOduration=1.7696880149999998 podStartE2EDuration="5.013115279s" totalImagesPullingTime="3.243427264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:59.012238179 +0000 UTC m=+63.917838720" watchObservedRunningTime="2026-08-04 19:16:59.013115279 +0000 UTC m=+63.918715817" Aug 04 19:16:59.030020 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:59.029969 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-69xrj" podStartSLOduration=1.248766442 podStartE2EDuration="3.029956753s" totalImagesPullingTime="1.781190311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:59.029511327 +0000 UTC m=+63.935111866" watchObservedRunningTime="2026-08-04 19:16:59.029956753 +0000 UTC m=+63.935557273" Aug 04 19:16:59.998811 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:16:59.998774 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mk4d7" event={"ID":"0784792e-39c0-4fa0-8f74-e3a016877e43","Type":"ContainerStarted","Data":"055e7967b7592b8787d8350e412cdb8f4800f4ec4ba39e7f8b6b4c79cd2771f9"} Aug 04 19:17:01.004086 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.004054 2580 generic.go:356] "Generic (PLEG): container finished" podID="a3608a9f-2b8b-4896-a64d-8b3eb9d38b04" containerID="5ffd45984d26ed8d95ec245772ec7ff179e88514ffae2cadc4699b76c6284ec1" exitCode=0 Aug 04 19:17:01.004500 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.004141 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04","Type":"ContainerDied","Data":"5ffd45984d26ed8d95ec245772ec7ff179e88514ffae2cadc4699b76c6284ec1"} Aug 04 19:17:01.006444 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.006422 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mk4d7" event={"ID":"0784792e-39c0-4fa0-8f74-e3a016877e43","Type":"ContainerStarted","Data":"594a5597e20b665cbb9114259ab607fa69d9ae02a7da801d8849ac887573303c"} Aug 04 19:17:01.059348 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.059303 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mk4d7" podStartSLOduration=16.858738307 podStartE2EDuration="20.059289868s" totalImagesPullingTime="3.200551561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:16:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:17:01.058535756 +0000 UTC m=+65.964136294" watchObservedRunningTime="2026-08-04 19:17:01.059289868 +0000 UTC m=+65.964890407" Aug 04 19:17:01.386620 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.386578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:17:01.386821 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.386628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:17:01.389681 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.389660 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 19:17:01.389750 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.389706 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 19:17:01.399267 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.399243 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/241e4560-d288-44ae-af3a-49be302f4d6a-original-pull-secret\") pod \"global-pull-secret-syncer-r9jp7\" (UID: \"241e4560-d288-44ae-af3a-49be302f4d6a\") " pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:17:01.399369 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.399277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6ee10d-1bcc-42fc-911d-76886bc6e292-metrics-certs\") pod \"network-metrics-daemon-t9kwv\" (UID: \"4e6ee10d-1bcc-42fc-911d-76886bc6e292\") " pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:17:01.487853 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.487819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46clc\" (UniqueName: \"kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc\") pod \"network-check-target-wk54h\" (UID: \"3747b048-0cd0-42b4-994f-4df2da8e8641\") " pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:17:01.490761 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.490743 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 19:17:01.500800 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.500779 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 19:17:01.511305 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.511279 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46clc\" (UniqueName: \"kubernetes.io/projected/3747b048-0cd0-42b4-994f-4df2da8e8641-kube-api-access-46clc\") pod \"network-check-target-wk54h\" (UID: \"3747b048-0cd0-42b4-994f-4df2da8e8641\") " pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:17:01.612995 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.612964 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bhjx\"" Aug 04 19:17:01.620234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.620211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:17:01.620357 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.620263 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pr7h8\"" Aug 04 19:17:01.625028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.625013 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9jp7" Aug 04 19:17:01.627610 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.627590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9kwv" Aug 04 19:17:01.757555 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.757459 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wk54h"] Aug 04 19:17:01.761134 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:17:01.761090 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3747b048_0cd0_42b4_994f_4df2da8e8641.slice/crio-177384ccde28b8935c68580103e6389533e297140e6b7643d555cf0e44fa0998 WatchSource:0}: Error finding container 177384ccde28b8935c68580103e6389533e297140e6b7643d555cf0e44fa0998: Status 404 returned error can't find the container with id 177384ccde28b8935c68580103e6389533e297140e6b7643d555cf0e44fa0998 Aug 04 19:17:01.778341 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.778315 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r9jp7"] Aug 04 19:17:01.780913 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:17:01.780886 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241e4560_d288_44ae_af3a_49be302f4d6a.slice/crio-55c143b1662817b2175b9cc6b2dc10d3731e744837a949e017be109b9f0e4aeb WatchSource:0}: Error finding container 55c143b1662817b2175b9cc6b2dc10d3731e744837a949e017be109b9f0e4aeb: Status 404 returned error can't find the container with id 55c143b1662817b2175b9cc6b2dc10d3731e744837a949e017be109b9f0e4aeb Aug 04 19:17:01.797392 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:01.797373 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t9kwv"] Aug 04 19:17:01.799560 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:17:01.799537 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6ee10d_1bcc_42fc_911d_76886bc6e292.slice/crio-d14c673ae01378a776c4f13ea04ac7a551352885ff50fa2e2c40cf189b4116a9 WatchSource:0}: Error finding container d14c673ae01378a776c4f13ea04ac7a551352885ff50fa2e2c40cf189b4116a9: Status 404 returned error can't find the container with id d14c673ae01378a776c4f13ea04ac7a551352885ff50fa2e2c40cf189b4116a9 Aug 04 19:17:02.011495 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:02.011446 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r9jp7" event={"ID":"241e4560-d288-44ae-af3a-49be302f4d6a","Type":"ContainerStarted","Data":"55c143b1662817b2175b9cc6b2dc10d3731e744837a949e017be109b9f0e4aeb"} Aug 04 19:17:02.012623 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:02.012602 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t9kwv" event={"ID":"4e6ee10d-1bcc-42fc-911d-76886bc6e292","Type":"ContainerStarted","Data":"d14c673ae01378a776c4f13ea04ac7a551352885ff50fa2e2c40cf189b4116a9"} Aug 04 19:17:02.013780 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:02.013757 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wk54h" event={"ID":"3747b048-0cd0-42b4-994f-4df2da8e8641","Type":"ContainerStarted","Data":"177384ccde28b8935c68580103e6389533e297140e6b7643d555cf0e44fa0998"} Aug 04 19:17:02.123683 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:02.123651 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:02.187002 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:02.186959 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:03.007327 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:03.007296 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:03.302018 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:03.301985 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:04.030032 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:04.029156 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t9kwv" event={"ID":"4e6ee10d-1bcc-42fc-911d-76886bc6e292","Type":"ContainerStarted","Data":"72a07021968340fc2ee12e52081834e82e39ae3317c4d905f8e743027520d068"} Aug 04 19:17:04.030032 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:04.029220 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t9kwv" event={"ID":"4e6ee10d-1bcc-42fc-911d-76886bc6e292","Type":"ContainerStarted","Data":"bd469f094873f552dfee04a7885900f244ff617938fe0ec7f0693c2d73e884da"} Aug 04 19:17:04.047128 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:04.047075 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t9kwv" podStartSLOduration=66.787610951 podStartE2EDuration="1m8.047055121s" totalImagesPullingTime="1.25944417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:17:04.045543534 +0000 UTC m=+68.951144074" watchObservedRunningTime="2026-08-04 19:17:04.047055121 +0000 UTC m=+68.952655661" Aug 04 19:17:04.048976 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:04.048934 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:04.147778 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:04.147743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:04.221564 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:04.221526 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:05.009586 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:05.009561 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b49478744-q9h8w" Aug 04 19:17:07.003183 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:07.003153 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d6c9b5f4c-n79qt"] Aug 04 19:17:07.103766 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:07.103731 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:07.312566 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:07.312481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:07.411382 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:07.411343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:09.058669 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.058631 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wk54h" event={"ID":"3747b048-0cd0-42b4-994f-4df2da8e8641","Type":"ContainerStarted","Data":"ddf45c6694da505fe9a1cf5ab0eeead12b52dbb684161a8c5ae4c7e4d848439d"} Aug 04 19:17:09.059098 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.058717 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:17:09.062048 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.062029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04","Type":"ContainerStarted","Data":"61b4715a51768baa1de136da1f54c95c0a2e18120d1d5bbf1d088622a80cafd9"} Aug 04 19:17:09.062179 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.062078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04","Type":"ContainerStarted","Data":"a75805646ede6f63c9e35a529fe266dd0bef1d8b998038d9430249b09e20c673"} Aug 04 19:17:09.062179 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.062114 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04","Type":"ContainerStarted","Data":"8e0ec375a5cfe73284a8c488043ed6da4f99cde05afa6a7fc8e10f8e90bbf439"} Aug 04 19:17:09.062179 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.062148 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04","Type":"ContainerStarted","Data":"394bdef1b150344240ac416a524800a5c856a4da8d6ae8496ba7a49cc12a425a"} Aug 04 19:17:09.062352 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.062185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04","Type":"ContainerStarted","Data":"689b99417af666d9895d9c30b6713c65732184932e9904b6ea00d23586424fc2"} Aug 04 19:17:09.062352 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.062213 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3608a9f-2b8b-4896-a64d-8b3eb9d38b04","Type":"ContainerStarted","Data":"9e813833204b44949ae5cdfd662c35a8da086b2695b9ac2cb834d2a2378624fb"} Aug 04 19:17:09.063528 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.063508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r9jp7" event={"ID":"241e4560-d288-44ae-af3a-49be302f4d6a","Type":"ContainerStarted","Data":"5496f3e84c546ebfd6c2c30327dc3995020d1631d995ffd0861d50d9324d073a"} Aug 04 19:17:09.075402 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.075366 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wk54h" podStartSLOduration=66.483578991 podStartE2EDuration="1m13.075354656s" totalImagesPullingTime="6.591775665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:17:09.075144343 +0000 UTC m=+73.980744883" watchObservedRunningTime="2026-08-04 19:17:09.075354656 +0000 UTC m=+73.980955206" Aug 04 19:17:09.108548 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.108499 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.575979083 podStartE2EDuration="12.10848097s" totalImagesPullingTime="8.509728218s" totalInitContainerRuntime="22.773669ms" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:17:09.107709844 +0000 UTC m=+74.013310394" watchObservedRunningTime="2026-08-04 19:17:09.10848097 +0000 UTC m=+74.014081541" Aug 04 19:17:09.123366 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:09.123324 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r9jp7" podStartSLOduration=66.541582045 podStartE2EDuration="1m13.123313418s" totalImagesPullingTime="6.581731373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:17:09.122316158 +0000 UTC m=+74.027916699" watchObservedRunningTime="2026-08-04 19:17:09.123313418 +0000 UTC m=+74.028913976" Aug 04 19:17:11.678029 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:11.677981 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:11.778237 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:11.778201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:13.153174 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:13.153135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:13.213909 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:13.213872 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:13.280771 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:13.280741 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:17:26.692853 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:26.692732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:26.793328 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:26.793297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:32.025149 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.025109 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d6c9b5f4c-n79qt" podUID="d3be89e3-d661-4b27-a5eb-7bd3df497e34" containerName="console" containerID="cri-o://952b498c8c700ab9a739dff81c61328a1acb01463506434872af7263f041a600" gracePeriod=15 Aug 04 19:17:32.158722 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.158693 2580 generic.go:356] "Generic (PLEG): container finished" podID="d3be89e3-d661-4b27-a5eb-7bd3df497e34" containerID="952b498c8c700ab9a739dff81c61328a1acb01463506434872af7263f041a600" exitCode=0 Aug 04 19:17:32.158880 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.158736 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6c9b5f4c-n79qt" event={"ID":"d3be89e3-d661-4b27-a5eb-7bd3df497e34","Type":"ContainerDied","Data":"952b498c8c700ab9a739dff81c61328a1acb01463506434872af7263f041a600"} Aug 04 19:17:32.273475 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.273448 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:17:32.325674 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.325602 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-serving-cert\") pod \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " Aug 04 19:17:32.325809 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.325691 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-service-ca\") pod \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " Aug 04 19:17:32.325809 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.325732 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-oauth-config\") pod \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " Aug 04 19:17:32.325809 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.325757 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-config\") pod \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " Aug 04 19:17:32.325809 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.325780 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8848z\" (UniqueName: \"kubernetes.io/projected/d3be89e3-d661-4b27-a5eb-7bd3df497e34-kube-api-access-8848z\") pod \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " Aug 04 19:17:32.326028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.325861 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-oauth-serving-cert\") pod \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " Aug 04 19:17:32.326028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.325909 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-trusted-ca-bundle\") pod \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\" (UID: \"d3be89e3-d661-4b27-a5eb-7bd3df497e34\") " Aug 04 19:17:32.326139 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.326090 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-service-ca" (OuterVolumeSpecName: "service-ca") pod "d3be89e3-d661-4b27-a5eb-7bd3df497e34" (UID: "d3be89e3-d661-4b27-a5eb-7bd3df497e34"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:17:32.326242 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.326214 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-config" (OuterVolumeSpecName: "console-config") pod "d3be89e3-d661-4b27-a5eb-7bd3df497e34" (UID: "d3be89e3-d661-4b27-a5eb-7bd3df497e34"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:17:32.326304 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.326271 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d3be89e3-d661-4b27-a5eb-7bd3df497e34" (UID: "d3be89e3-d661-4b27-a5eb-7bd3df497e34"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:17:32.326521 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.326500 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d3be89e3-d661-4b27-a5eb-7bd3df497e34" (UID: "d3be89e3-d661-4b27-a5eb-7bd3df497e34"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:17:32.328148 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.328128 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3be89e3-d661-4b27-a5eb-7bd3df497e34-kube-api-access-8848z" (OuterVolumeSpecName: "kube-api-access-8848z") pod "d3be89e3-d661-4b27-a5eb-7bd3df497e34" (UID: "d3be89e3-d661-4b27-a5eb-7bd3df497e34"). InnerVolumeSpecName "kube-api-access-8848z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:17:32.328437 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.328417 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d3be89e3-d661-4b27-a5eb-7bd3df497e34" (UID: "d3be89e3-d661-4b27-a5eb-7bd3df497e34"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:17:32.328530 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.328496 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d3be89e3-d661-4b27-a5eb-7bd3df497e34" (UID: "d3be89e3-d661-4b27-a5eb-7bd3df497e34"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:17:32.426780 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.426754 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-oauth-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:17:32.426780 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.426777 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:17:32.426963 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.426789 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8848z\" (UniqueName: \"kubernetes.io/projected/d3be89e3-d661-4b27-a5eb-7bd3df497e34-kube-api-access-8848z\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:17:32.426963 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.426802 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-oauth-serving-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:17:32.426963 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.426810 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-trusted-ca-bundle\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:17:32.426963 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.426818 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3be89e3-d661-4b27-a5eb-7bd3df497e34-console-serving-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:17:32.426963 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:32.426826 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3be89e3-d661-4b27-a5eb-7bd3df497e34-service-ca\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:17:33.163150 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:33.163125 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6c9b5f4c-n79qt" Aug 04 19:17:33.163150 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:33.163136 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6c9b5f4c-n79qt" event={"ID":"d3be89e3-d661-4b27-a5eb-7bd3df497e34","Type":"ContainerDied","Data":"582a532030e24c43a9ae18294c0b66d35aca2a4270bb5c3052f30cdf775aed93"} Aug 04 19:17:33.163684 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:33.163195 2580 scope.go:117] "RemoveContainer" containerID="952b498c8c700ab9a739dff81c61328a1acb01463506434872af7263f041a600" Aug 04 19:17:33.184285 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:33.184262 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d6c9b5f4c-n79qt"] Aug 04 19:17:33.187839 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:33.187817 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d6c9b5f4c-n79qt"] Aug 04 19:17:33.602210 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:33.602176 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3be89e3-d661-4b27-a5eb-7bd3df497e34" path="/var/lib/kubelet/pods/d3be89e3-d661-4b27-a5eb-7bd3df497e34/volumes" Aug 04 19:17:35.910131 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:35.910078 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:36.279097 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:36.279064 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:36.394726 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:36.394696 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:17:40.070242 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:40.070210 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wk54h" Aug 04 19:17:58.280486 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:58.280443 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:17:58.300027 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:58.299997 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:17:59.271685 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:17:59.271657 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 19:18:10.964008 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:10.963974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:11.212884 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:11.212845 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:11.414247 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:11.414146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:11.478816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:11.478781 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:18.383209 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:18.383182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:18.441266 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:18.441239 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:18.542434 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:18.542405 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:18.599544 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:18.599515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:19.416739 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:19.416674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:19.469000 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:19.468974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:26.253538 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:26.253494 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:26.371305 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:26.371269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:30.129809 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:30.129781 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:38.133967 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:38.133925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:38.188515 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:38.188452 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:40.442356 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:40.442319 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:40.537701 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:40.537669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:40.637819 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:40.637786 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:41.686229 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:41.686201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:41.784089 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:41.784063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:18:59.370544 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:18:59.370453 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:03.490967 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:03.490934 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:10.398700 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:10.398667 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:10.497253 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:10.497219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:42.194378 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:42.194347 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:43.049425 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:43.049386 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:46.012802 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:46.012763 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:46.110222 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:46.110185 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:47.123297 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:47.123253 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:47.328931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:47.328897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:47.910224 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:47.910186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:48.025867 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:48.025838 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:49.895219 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:49.895189 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:50.294869 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:50.294840 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:52.790922 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:52.790890 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:52.854419 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:52.854388 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:57.960844 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:57.960807 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:58.024248 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.024220 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:58.494217 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.494183 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:19:58.794145 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.794071 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5xhtm"] Aug 04 19:19:58.794367 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.794355 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3be89e3-d661-4b27-a5eb-7bd3df497e34" containerName="console" Aug 04 19:19:58.794409 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.794369 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3be89e3-d661-4b27-a5eb-7bd3df497e34" containerName="console" Aug 04 19:19:58.794451 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.794411 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3be89e3-d661-4b27-a5eb-7bd3df497e34" containerName="console" Aug 04 19:19:58.796540 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.796526 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5xhtm" Aug 04 19:19:58.799024 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.798999 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 19:19:58.799024 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.799017 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 19:19:58.799949 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.799936 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mk5th\"" Aug 04 19:19:58.806631 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.806611 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5xhtm"] Aug 04 19:19:58.820909 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.820883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce874ab4-1087-4ef0-b9ee-c626aee38618-bound-sa-token\") pod \"cert-manager-55b4d954d5-5xhtm\" (UID: \"ce874ab4-1087-4ef0-b9ee-c626aee38618\") " pod="cert-manager/cert-manager-55b4d954d5-5xhtm" Aug 04 19:19:58.821007 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.820923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrfl\" (UniqueName: \"kubernetes.io/projected/ce874ab4-1087-4ef0-b9ee-c626aee38618-kube-api-access-mzrfl\") pod \"cert-manager-55b4d954d5-5xhtm\" (UID: \"ce874ab4-1087-4ef0-b9ee-c626aee38618\") " pod="cert-manager/cert-manager-55b4d954d5-5xhtm" Aug 04 19:19:58.922152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.922119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrfl\" (UniqueName: \"kubernetes.io/projected/ce874ab4-1087-4ef0-b9ee-c626aee38618-kube-api-access-mzrfl\") pod \"cert-manager-55b4d954d5-5xhtm\" (UID: \"ce874ab4-1087-4ef0-b9ee-c626aee38618\") " pod="cert-manager/cert-manager-55b4d954d5-5xhtm" Aug 04 19:19:58.922301 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.922173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce874ab4-1087-4ef0-b9ee-c626aee38618-bound-sa-token\") pod \"cert-manager-55b4d954d5-5xhtm\" (UID: \"ce874ab4-1087-4ef0-b9ee-c626aee38618\") " pod="cert-manager/cert-manager-55b4d954d5-5xhtm" Aug 04 19:19:58.930209 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.930185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce874ab4-1087-4ef0-b9ee-c626aee38618-bound-sa-token\") pod \"cert-manager-55b4d954d5-5xhtm\" (UID: \"ce874ab4-1087-4ef0-b9ee-c626aee38618\") " pod="cert-manager/cert-manager-55b4d954d5-5xhtm" Aug 04 19:19:58.930360 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:58.930342 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrfl\" (UniqueName: \"kubernetes.io/projected/ce874ab4-1087-4ef0-b9ee-c626aee38618-kube-api-access-mzrfl\") pod \"cert-manager-55b4d954d5-5xhtm\" (UID: \"ce874ab4-1087-4ef0-b9ee-c626aee38618\") " pod="cert-manager/cert-manager-55b4d954d5-5xhtm" Aug 04 19:19:59.118813 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:59.118747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5xhtm" Aug 04 19:19:59.241571 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:59.241486 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5xhtm"] Aug 04 19:19:59.244085 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:19:59.244046 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce874ab4_1087_4ef0_b9ee_c626aee38618.slice/crio-5b592e915cc9197012f5e6711d5fac1d8525b47e1064767112c4de4d7c9ac872 WatchSource:0}: Error finding container 5b592e915cc9197012f5e6711d5fac1d8525b47e1064767112c4de4d7c9ac872: Status 404 returned error can't find the container with id 5b592e915cc9197012f5e6711d5fac1d8525b47e1064767112c4de4d7c9ac872 Aug 04 19:19:59.681674 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:19:59.681642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5xhtm" event={"ID":"ce874ab4-1087-4ef0-b9ee-c626aee38618","Type":"ContainerStarted","Data":"5b592e915cc9197012f5e6711d5fac1d8525b47e1064767112c4de4d7c9ac872"} Aug 04 19:20:00.293408 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:00.293369 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:00.378494 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:00.378423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:02.696284 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:02.696257 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5xhtm" event={"ID":"ce874ab4-1087-4ef0-b9ee-c626aee38618","Type":"ContainerStarted","Data":"6976e6043cf1810b5d48b83ec07081030253992125447544eb31dc3ab7bc43da"} Aug 04 19:20:02.711601 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:02.711553 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-5xhtm" podStartSLOduration=1.326116751 podStartE2EDuration="4.711537394s" totalImagesPullingTime="3.385420643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:02.71152357 +0000 UTC m=+247.617124112" watchObservedRunningTime="2026-08-04 19:20:02.711537394 +0000 UTC m=+247.617137932" Aug 04 19:20:06.437618 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:06.437584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:06.534826 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:06.534775 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:08.663623 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:08.663590 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:15.079750 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:15.079714 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:21.227829 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:21.227740 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:21.318948 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:21.318914 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:21.848367 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:21.848330 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:22.057810 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.057784 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:22.153574 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.153491 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:22.203691 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.203670 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92"] Aug 04 19:20:22.205739 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.205723 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.210778 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.210755 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 19:20:22.210897 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.210777 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 19:20:22.210897 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.210829 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9bp4b\"" Aug 04 19:20:22.211097 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.211069 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 19:20:22.211158 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.211139 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 19:20:22.226112 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.226088 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92"] Aug 04 19:20:22.300569 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.300545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2bf84db8-d140-4fdb-a0f8-2fee781b6530-manifests\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.300840 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.300573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2bf84db8-d140-4fdb-a0f8-2fee781b6530-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.300840 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.300615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bf84db8-d140-4fdb-a0f8-2fee781b6530-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.300840 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.300660 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bf84db8-d140-4fdb-a0f8-2fee781b6530-tmp\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.300840 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.300677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkw6h\" (UniqueName: \"kubernetes.io/projected/2bf84db8-d140-4fdb-a0f8-2fee781b6530-kube-api-access-bkw6h\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.332128 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.332109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:22.402027 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.401998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bf84db8-d140-4fdb-a0f8-2fee781b6530-tmp\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.402143 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.402031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkw6h\" (UniqueName: \"kubernetes.io/projected/2bf84db8-d140-4fdb-a0f8-2fee781b6530-kube-api-access-bkw6h\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.402143 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.402063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2bf84db8-d140-4fdb-a0f8-2fee781b6530-manifests\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.402143 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.402084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2bf84db8-d140-4fdb-a0f8-2fee781b6530-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.402143 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.402138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bf84db8-d140-4fdb-a0f8-2fee781b6530-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.402556 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.402530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2bf84db8-d140-4fdb-a0f8-2fee781b6530-manifests\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.404483 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.404413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bf84db8-d140-4fdb-a0f8-2fee781b6530-tmp\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.404701 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.404681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bf84db8-d140-4fdb-a0f8-2fee781b6530-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.404766 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.404702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2bf84db8-d140-4fdb-a0f8-2fee781b6530-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.413157 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.413135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkw6h\" (UniqueName: \"kubernetes.io/projected/2bf84db8-d140-4fdb-a0f8-2fee781b6530-kube-api-access-bkw6h\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-bmz92\" (UID: \"2bf84db8-d140-4fdb-a0f8-2fee781b6530\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.431731 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.431709 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:22.514826 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.514803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:22.531542 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.531518 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:22.632285 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.632249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:22.652365 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.652340 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92"] Aug 04 19:20:22.655677 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:20:22.655627 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf84db8_d140_4fdb_a0f8_2fee781b6530.slice/crio-9ce8b89e70a7e78017a31a14c426001d8250ba9315e213b9f1bb7e1eae9e966d WatchSource:0}: Error finding container 9ce8b89e70a7e78017a31a14c426001d8250ba9315e213b9f1bb7e1eae9e966d: Status 404 returned error can't find the container with id 9ce8b89e70a7e78017a31a14c426001d8250ba9315e213b9f1bb7e1eae9e966d Aug 04 19:20:22.738015 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.737993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:22.764137 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.764114 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" event={"ID":"2bf84db8-d140-4fdb-a0f8-2fee781b6530","Type":"ContainerStarted","Data":"9ce8b89e70a7e78017a31a14c426001d8250ba9315e213b9f1bb7e1eae9e966d"} Aug 04 19:20:22.832086 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.832057 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:22.933617 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:22.933553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:25.778982 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:25.778951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" event={"ID":"2bf84db8-d140-4fdb-a0f8-2fee781b6530","Type":"ContainerStarted","Data":"92d72744840fe1873474845b8ed2a7ffc254cfbced03c98f86cf391878aeb5cb"} Aug 04 19:20:27.789050 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:27.789012 2580 generic.go:356] "Generic (PLEG): container finished" podID="2bf84db8-d140-4fdb-a0f8-2fee781b6530" containerID="92d72744840fe1873474845b8ed2a7ffc254cfbced03c98f86cf391878aeb5cb" exitCode=0 Aug 04 19:20:27.789050 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:27.789041 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" event={"ID":"2bf84db8-d140-4fdb-a0f8-2fee781b6530","Type":"ContainerDied","Data":"92d72744840fe1873474845b8ed2a7ffc254cfbced03c98f86cf391878aeb5cb"} Aug 04 19:20:28.794927 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:28.794895 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" event={"ID":"2bf84db8-d140-4fdb-a0f8-2fee781b6530","Type":"ContainerStarted","Data":"2812b1a4a3a0f344789f3ba77ec985753cf0ad70388eaba26d9056246e93a9c8"} Aug 04 19:20:28.795306 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:28.795105 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:28.818951 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:28.818903 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" podStartSLOduration=2.338053087 podStartE2EDuration="6.818888094s" totalImagesPullingTime="3.455981336s" totalInitContainerRuntime="1.024853671s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:28.817005225 +0000 UTC m=+273.722605766" watchObservedRunningTime="2026-08-04 19:20:28.818888094 +0000 UTC m=+273.724488633" Aug 04 19:20:34.239440 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:34.239406 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:36.222808 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:36.222771 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:36.371919 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:36.371891 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:36.471950 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:36.471919 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:39.801482 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:39.801430 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-bmz92" Aug 04 19:20:41.328710 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:41.328678 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:41.417924 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:41.417839 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:42.445791 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:42.445748 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:42.547771 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:42.547732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:42.645240 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:42.645207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:42.756903 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:42.756811 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:43.134108 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.134023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:43.183716 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.183686 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rtd5m"] Aug 04 19:20:43.191392 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.191370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" Aug 04 19:20:43.194487 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.194446 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pmqkr\"" Aug 04 19:20:43.194610 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.194542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Aug 04 19:20:43.201933 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.201900 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rtd5m"] Aug 04 19:20:43.273641 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.273610 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:43.368244 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.368203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648eac31-d4cb-4714-9ef9-fe3f3687fb58-tls-certs\") pod \"model-serving-api-54d89bd5fc-rtd5m\" (UID: \"648eac31-d4cb-4714-9ef9-fe3f3687fb58\") " pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" Aug 04 19:20:43.368416 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.368327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpwbh\" (UniqueName: \"kubernetes.io/projected/648eac31-d4cb-4714-9ef9-fe3f3687fb58-kube-api-access-kpwbh\") pod \"model-serving-api-54d89bd5fc-rtd5m\" (UID: \"648eac31-d4cb-4714-9ef9-fe3f3687fb58\") " pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" Aug 04 19:20:43.469246 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.469215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648eac31-d4cb-4714-9ef9-fe3f3687fb58-tls-certs\") pod \"model-serving-api-54d89bd5fc-rtd5m\" (UID: \"648eac31-d4cb-4714-9ef9-fe3f3687fb58\") " pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" Aug 04 19:20:43.469617 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.469331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpwbh\" (UniqueName: \"kubernetes.io/projected/648eac31-d4cb-4714-9ef9-fe3f3687fb58-kube-api-access-kpwbh\") pod \"model-serving-api-54d89bd5fc-rtd5m\" (UID: \"648eac31-d4cb-4714-9ef9-fe3f3687fb58\") " pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" Aug 04 19:20:43.471667 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.471645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648eac31-d4cb-4714-9ef9-fe3f3687fb58-tls-certs\") pod \"model-serving-api-54d89bd5fc-rtd5m\" (UID: \"648eac31-d4cb-4714-9ef9-fe3f3687fb58\") " pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" Aug 04 19:20:43.480524 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.480502 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpwbh\" (UniqueName: \"kubernetes.io/projected/648eac31-d4cb-4714-9ef9-fe3f3687fb58-kube-api-access-kpwbh\") pod \"model-serving-api-54d89bd5fc-rtd5m\" (UID: \"648eac31-d4cb-4714-9ef9-fe3f3687fb58\") " pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" Aug 04 19:20:43.502010 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.501985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" Aug 04 19:20:43.641603 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.641581 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rtd5m"] Aug 04 19:20:43.643539 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:20:43.643513 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648eac31_d4cb_4714_9ef9_fe3f3687fb58.slice/crio-5c60a4ae3ad8b0136ee3f61bf9df6612f8b54f8df1a943c3628960008a72d543 WatchSource:0}: Error finding container 5c60a4ae3ad8b0136ee3f61bf9df6612f8b54f8df1a943c3628960008a72d543: Status 404 returned error can't find the container with id 5c60a4ae3ad8b0136ee3f61bf9df6612f8b54f8df1a943c3628960008a72d543 Aug 04 19:20:43.853871 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:43.853782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" event={"ID":"648eac31-d4cb-4714-9ef9-fe3f3687fb58","Type":"ContainerStarted","Data":"5c60a4ae3ad8b0136ee3f61bf9df6612f8b54f8df1a943c3628960008a72d543"} Aug 04 19:20:46.867628 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:46.867543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" event={"ID":"648eac31-d4cb-4714-9ef9-fe3f3687fb58","Type":"ContainerStarted","Data":"07eec52e55bdf4386b6b6958cfe6749b861d63a996161ce3080f22068a9090ee"} Aug 04 19:20:46.868040 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:46.867661 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" Aug 04 19:20:47.696313 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.696283 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:47.808691 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.808642 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" podStartSLOduration=1.908685664 podStartE2EDuration="4.808625432s" totalImagesPullingTime="2.899939768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:46.896374287 +0000 UTC m=+291.801974826" watchObservedRunningTime="2026-08-04 19:20:47.808625432 +0000 UTC m=+292.714225973" Aug 04 19:20:47.809165 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.809149 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn"] Aug 04 19:20:47.812643 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.812622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:47.817659 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.817643 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 19:20:47.820792 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.820769 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2l9m2\"" Aug 04 19:20:47.821106 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.821092 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 19:20:47.821662 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.821650 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 19:20:47.822331 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.822317 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 19:20:47.897448 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.897424 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn"] Aug 04 19:20:47.910660 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.910445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kxbl\" (UniqueName: \"kubernetes.io/projected/6c49c0c7-58e9-491c-9065-44930435bab9-kube-api-access-7kxbl\") pod \"kube-auth-proxy-7fcb8f55f4-rnxjn\" (UID: \"6c49c0c7-58e9-491c-9065-44930435bab9\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:47.910660 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.910518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c49c0c7-58e9-491c-9065-44930435bab9-tls-certs\") pod \"kube-auth-proxy-7fcb8f55f4-rnxjn\" (UID: \"6c49c0c7-58e9-491c-9065-44930435bab9\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:47.910660 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.910583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c49c0c7-58e9-491c-9065-44930435bab9-tmp\") pod \"kube-auth-proxy-7fcb8f55f4-rnxjn\" (UID: \"6c49c0c7-58e9-491c-9065-44930435bab9\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:47.948189 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:47.948116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:48.011318 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:48.011293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c49c0c7-58e9-491c-9065-44930435bab9-tls-certs\") pod \"kube-auth-proxy-7fcb8f55f4-rnxjn\" (UID: \"6c49c0c7-58e9-491c-9065-44930435bab9\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:48.011440 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:48.011338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c49c0c7-58e9-491c-9065-44930435bab9-tmp\") pod \"kube-auth-proxy-7fcb8f55f4-rnxjn\" (UID: \"6c49c0c7-58e9-491c-9065-44930435bab9\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:48.011440 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:48.011372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kxbl\" (UniqueName: \"kubernetes.io/projected/6c49c0c7-58e9-491c-9065-44930435bab9-kube-api-access-7kxbl\") pod \"kube-auth-proxy-7fcb8f55f4-rnxjn\" (UID: \"6c49c0c7-58e9-491c-9065-44930435bab9\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:48.013774 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:48.013750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c49c0c7-58e9-491c-9065-44930435bab9-tmp\") pod \"kube-auth-proxy-7fcb8f55f4-rnxjn\" (UID: \"6c49c0c7-58e9-491c-9065-44930435bab9\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:48.013906 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:48.013888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c49c0c7-58e9-491c-9065-44930435bab9-tls-certs\") pod \"kube-auth-proxy-7fcb8f55f4-rnxjn\" (UID: \"6c49c0c7-58e9-491c-9065-44930435bab9\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:48.022157 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:48.022137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kxbl\" (UniqueName: \"kubernetes.io/projected/6c49c0c7-58e9-491c-9065-44930435bab9-kube-api-access-7kxbl\") pod \"kube-auth-proxy-7fcb8f55f4-rnxjn\" (UID: \"6c49c0c7-58e9-491c-9065-44930435bab9\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:48.122242 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:48.122221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" Aug 04 19:20:48.264159 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:48.264137 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn"] Aug 04 19:20:48.266537 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:20:48.266514 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c49c0c7_58e9_491c_9065_44930435bab9.slice/crio-7aad4965a623bbd0289858e5c8ed0ebfc5a5f2abe358585eb49848654e406c16 WatchSource:0}: Error finding container 7aad4965a623bbd0289858e5c8ed0ebfc5a5f2abe358585eb49848654e406c16: Status 404 returned error can't find the container with id 7aad4965a623bbd0289858e5c8ed0ebfc5a5f2abe358585eb49848654e406c16 Aug 04 19:20:48.875836 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:48.875807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" event={"ID":"6c49c0c7-58e9-491c-9065-44930435bab9","Type":"ContainerStarted","Data":"7aad4965a623bbd0289858e5c8ed0ebfc5a5f2abe358585eb49848654e406c16"} Aug 04 19:20:49.363694 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.363660 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q"] Aug 04 19:20:49.366946 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.366923 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.369848 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.369825 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 19:20:49.371133 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.371109 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 19:20:49.371222 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.371157 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xf98l\"" Aug 04 19:20:49.371222 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.371195 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 19:20:49.371222 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.371207 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:20:49.371528 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.371511 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 19:20:49.384435 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.384414 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q"] Aug 04 19:20:49.422934 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.422906 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-manager-config\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.423027 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.422945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4tt\" (UniqueName: \"kubernetes.io/projected/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-kube-api-access-vn4tt\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.423083 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.423028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-cert\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.423083 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.423054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-metrics-cert\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.524204 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.524181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4tt\" (UniqueName: \"kubernetes.io/projected/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-kube-api-access-vn4tt\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.524286 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.524240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-cert\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.524286 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.524257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-metrics-cert\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.524286 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.524276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-manager-config\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.524848 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.524829 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-manager-config\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.526904 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.526884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-metrics-cert\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.526981 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.526958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-cert\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.538316 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.538290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4tt\" (UniqueName: \"kubernetes.io/projected/25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6-kube-api-access-vn4tt\") pod \"lws-controller-manager-59d56c454c-gcq5q\" (UID: \"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.676922 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.676854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:49.826843 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.826811 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q"] Aug 04 19:20:49.829553 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:20:49.829533 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e3e5d5_e8e1_4bf6_917a_5fdb9a8897c6.slice/crio-3fad183eb3eaa6232cf48c1ee66f394a2c7ba3cb98d897ebebfd169c6acecb47 WatchSource:0}: Error finding container 3fad183eb3eaa6232cf48c1ee66f394a2c7ba3cb98d897ebebfd169c6acecb47: Status 404 returned error can't find the container with id 3fad183eb3eaa6232cf48c1ee66f394a2c7ba3cb98d897ebebfd169c6acecb47 Aug 04 19:20:49.880738 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:49.880707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" event={"ID":"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6","Type":"ContainerStarted","Data":"3fad183eb3eaa6232cf48c1ee66f394a2c7ba3cb98d897ebebfd169c6acecb47"} Aug 04 19:20:50.026503 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:50.026446 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:50.137358 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:50.137328 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:50.231488 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:50.231438 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:51.891481 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:51.891376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" event={"ID":"6c49c0c7-58e9-491c-9065-44930435bab9","Type":"ContainerStarted","Data":"5ca2ff8f83b9278e2f226665c84bb83e9e04ecf69cc4e0fc73f557c827d085a9"} Aug 04 19:20:51.935914 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:51.935863 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-rnxjn" podStartSLOduration=1.9223390710000001 podStartE2EDuration="4.935848182s" totalImagesPullingTime="3.013509111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:51.93234716 +0000 UTC m=+296.837947700" watchObservedRunningTime="2026-08-04 19:20:51.935848182 +0000 UTC m=+296.841448722" Aug 04 19:20:52.794447 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:52.794395 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:53.901633 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:53.901596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" event={"ID":"25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6","Type":"ContainerStarted","Data":"b169aa5cae9691706c4aed1149a1ce1639a89ebd980e0d4ff34f96e72ca46c23"} Aug 04 19:20:53.902009 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:53.901768 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:20:53.939761 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:53.939714 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" podStartSLOduration=1.7196877229999998 podStartE2EDuration="4.939699353s" totalImagesPullingTime="3.22001163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:53.938637812 +0000 UTC m=+298.844238350" watchObservedRunningTime="2026-08-04 19:20:53.939699353 +0000 UTC m=+298.845299956" Aug 04 19:20:54.054998 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:54.054949 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:54.192679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:54.192645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:54.330243 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:54.330208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:54.434203 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:54.434156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:54.568581 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:54.568498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:55.452966 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:55.452933 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:20:55.454128 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:55.454107 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:20:55.458259 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:55.458240 2580 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 19:20:55.607549 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:55.607361 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:55.713890 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:55.713802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:55.821808 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:55.821766 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:56.201922 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:56.201883 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:56.304720 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:56.304694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:57.875075 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:57.875050 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-rtd5m" Aug 04 19:20:57.964325 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:57.964293 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:58.232615 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:58.232580 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:20:58.870098 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:20:58.870062 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:21:04.908669 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:21:04.908638 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-gcq5q" Aug 04 19:21:43.016571 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:21:43.016532 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:21:43.102514 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:21:43.102483 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:21:58.735588 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:21:58.735483 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:01.544388 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:01.544356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:03.328058 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:03.328025 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:03.564029 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:03.563992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:03.709059 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:03.709020 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:04.322605 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:04.322578 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:04.778439 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:04.778407 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:05.941329 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:05.941298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:06.044271 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:06.044243 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:07.541238 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:07.541208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:07.957260 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:07.957228 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:09.819417 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:09.819385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:09.922755 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:09.922719 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:11.333170 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:11.333130 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:11.439509 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:11.439482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:13.327326 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:13.327292 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:13.382862 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:13.382837 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:14.788675 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.788642 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:14.796801 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.796778 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t"] Aug 04 19:22:14.802502 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.800272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:14.803288 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.803258 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Aug 04 19:22:14.803406 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.803330 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Aug 04 19:22:14.803406 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.803342 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xpfq2\"" Aug 04 19:22:14.804433 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.804408 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 19:22:14.804533 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.804506 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 19:22:14.808613 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.808592 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t"] Aug 04 19:22:14.878589 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.878566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zr42\" (UniqueName: \"kubernetes.io/projected/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-kube-api-access-7zr42\") pod \"kuadrant-console-plugin-598f86468b-8jp4t\" (UID: \"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:14.878705 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.878678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-8jp4t\" (UID: \"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:14.878761 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.878710 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-8jp4t\" (UID: \"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:14.979371 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.979346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-8jp4t\" (UID: \"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:14.979516 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.979385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-8jp4t\" (UID: \"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:14.979516 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.979413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zr42\" (UniqueName: \"kubernetes.io/projected/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-kube-api-access-7zr42\") pod \"kuadrant-console-plugin-598f86468b-8jp4t\" (UID: \"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:14.979590 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:22:14.979516 2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Aug 04 19:22:14.979590 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:22:14.979572 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-plugin-serving-cert podName:0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71 nodeName:}" failed. No retries permitted until 2026-08-04 19:22:15.479557127 +0000 UTC m=+380.385157644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-8jp4t" (UID: "0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71") : secret "plugin-serving-cert" not found Aug 04 19:22:14.980054 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.980037 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-8jp4t\" (UID: \"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:14.988950 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:14.988928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zr42\" (UniqueName: \"kubernetes.io/projected/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-kube-api-access-7zr42\") pod \"kuadrant-console-plugin-598f86468b-8jp4t\" (UID: \"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:15.191189 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:15.191150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:15.296373 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:15.296341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:15.483148 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:15.483057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-8jp4t\" (UID: \"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:15.485538 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:15.485515 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-8jp4t\" (UID: \"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:15.711540 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:15.711504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" Aug 04 19:22:15.855648 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:15.855616 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t"] Aug 04 19:22:15.859507 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:22:15.859481 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ce3c2c1_5ba6_4b5f_9c2e_a83659f9da71.slice/crio-5d616c8b80cdf469c9a68c4558ad6de7b7efaf8dda629bd5d3b2093e22baac35 WatchSource:0}: Error finding container 5d616c8b80cdf469c9a68c4558ad6de7b7efaf8dda629bd5d3b2093e22baac35: Status 404 returned error can't find the container with id 5d616c8b80cdf469c9a68c4558ad6de7b7efaf8dda629bd5d3b2093e22baac35 Aug 04 19:22:15.861325 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:15.861305 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:22:16.242651 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:16.242622 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" event={"ID":"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71","Type":"ContainerStarted","Data":"5d616c8b80cdf469c9a68c4558ad6de7b7efaf8dda629bd5d3b2093e22baac35"} Aug 04 19:22:18.036141 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:18.036111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:19.192560 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:19.192513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:19.291389 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:19.291342 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:19.388241 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:19.388195 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:21.032929 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:21.032887 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:21.093015 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:21.092983 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:22.271079 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:22.271050 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" event={"ID":"0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71","Type":"ContainerStarted","Data":"0ff27e7ce3a2f535366c0f8719080602cee1ece56da37e4a801f6297c093f70c"} Aug 04 19:22:22.286971 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:22.286928 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8jp4t" podStartSLOduration=2.69661633 podStartE2EDuration="8.286913947s" totalImagesPullingTime="5.590297617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:22:22.285924774 +0000 UTC m=+387.191525314" watchObservedRunningTime="2026-08-04 19:22:22.286913947 +0000 UTC m=+387.192514485" Aug 04 19:22:25.001213 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.001172 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5"] Aug 04 19:22:25.004021 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.004000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:25.006999 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.006982 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-846sv\"" Aug 04 19:22:25.031044 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.031020 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5"] Aug 04 19:22:25.062958 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.062936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ss5x5\" (UID: \"8cf419f0-02db-41c6-8fd0-3f3fee7ebace\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:25.063041 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.062966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pmst\" (UniqueName: \"kubernetes.io/projected/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-kube-api-access-4pmst\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ss5x5\" (UID: \"8cf419f0-02db-41c6-8fd0-3f3fee7ebace\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:25.163976 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.163944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ss5x5\" (UID: \"8cf419f0-02db-41c6-8fd0-3f3fee7ebace\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:25.164121 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.163988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pmst\" (UniqueName: \"kubernetes.io/projected/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-kube-api-access-4pmst\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ss5x5\" (UID: \"8cf419f0-02db-41c6-8fd0-3f3fee7ebace\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:25.164292 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.164270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ss5x5\" (UID: \"8cf419f0-02db-41c6-8fd0-3f3fee7ebace\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:25.175623 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.175594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pmst\" (UniqueName: \"kubernetes.io/projected/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-kube-api-access-4pmst\") pod \"kuadrant-operator-controller-manager-6bd956ccf-ss5x5\" (UID: \"8cf419f0-02db-41c6-8fd0-3f3fee7ebace\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:25.185891 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.185868 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5"] Aug 04 19:22:25.186121 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.186109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:25.262641 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.262579 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5"] Aug 04 19:22:25.280401 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:25.280368 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5"] Aug 04 19:22:25.448871 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:22:25.448836 2580 log.go:32] "RunPodSandbox from runtime service failed" err=< Aug 04 19:22:25.448871 ip-10-0-136-99 kubenswrapper[2580]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6bd956ccf-ss5x5_kuadrant-system_8cf419f0-02db-41c6-8fd0-3f3fee7ebace_0(3eee5a872391a3d6b3383cb0b480e09686e07f558ee4936ba22ead1f70012954): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6bd956ccf-ss5x5 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3eee5a872391a3d6b3383cb0b480e09686e07f558ee4936ba22ead1f70012954" Netns:"/var/run/netns/fdef8e14-2f61-41ac-b6ba-7a121952d5cf" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-6bd956ccf-ss5x5;K8S_POD_INFRA_CONTAINER_ID=3eee5a872391a3d6b3383cb0b480e09686e07f558ee4936ba22ead1f70012954;K8S_POD_UID=8cf419f0-02db-41c6-8fd0-3f3fee7ebace" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5] networking: [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5/8cf419f0-02db-41c6-8fd0-3f3fee7ebace:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5 3eee5a872391a3d6b3383cb0b480e09686e07f558ee4936ba22ead1f70012954 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:16 [10.134.0.22/23] Aug 04 19:22:25.448871 ip-10-0-136-99 kubenswrapper[2580]: ' Aug 04 19:22:25.448871 ip-10-0-136-99 kubenswrapper[2580]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Aug 04 19:22:25.448871 ip-10-0-136-99 kubenswrapper[2580]: > Aug 04 19:22:25.449064 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:22:25.448899 2580 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Aug 04 19:22:25.449064 ip-10-0-136-99 kubenswrapper[2580]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6bd956ccf-ss5x5_kuadrant-system_8cf419f0-02db-41c6-8fd0-3f3fee7ebace_0(3eee5a872391a3d6b3383cb0b480e09686e07f558ee4936ba22ead1f70012954): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6bd956ccf-ss5x5 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3eee5a872391a3d6b3383cb0b480e09686e07f558ee4936ba22ead1f70012954" Netns:"/var/run/netns/fdef8e14-2f61-41ac-b6ba-7a121952d5cf" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-6bd956ccf-ss5x5;K8S_POD_INFRA_CONTAINER_ID=3eee5a872391a3d6b3383cb0b480e09686e07f558ee4936ba22ead1f70012954;K8S_POD_UID=8cf419f0-02db-41c6-8fd0-3f3fee7ebace" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5] networking: [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5/8cf419f0-02db-41c6-8fd0-3f3fee7ebace:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5 3eee5a872391a3d6b3383cb0b480e09686e07f558ee4936ba22ead1f70012954 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:16 [10.134.0.22/23] Aug 04 19:22:25.449064 ip-10-0-136-99 kubenswrapper[2580]: ' Aug 04 19:22:25.449064 ip-10-0-136-99 kubenswrapper[2580]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Aug 04 19:22:25.449064 ip-10-0-136-99 kubenswrapper[2580]: > pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:26.288244 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:26.288216 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:26.294655 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:26.294634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:26.301714 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:22:26.301690 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-ss5x5\" is forbidden: User \"system:node:ip-10-0-136-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-99.ec2.internal' and this object" podUID="8cf419f0-02db-41c6-8fd0-3f3fee7ebace" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:26.374073 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:26.374049 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-extensions-socket-volume\") pod \"8cf419f0-02db-41c6-8fd0-3f3fee7ebace\" (UID: \"8cf419f0-02db-41c6-8fd0-3f3fee7ebace\") " Aug 04 19:22:26.374163 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:26.374122 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pmst\" (UniqueName: \"kubernetes.io/projected/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-kube-api-access-4pmst\") pod \"8cf419f0-02db-41c6-8fd0-3f3fee7ebace\" (UID: \"8cf419f0-02db-41c6-8fd0-3f3fee7ebace\") " Aug 04 19:22:26.374277 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:26.374254 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8cf419f0-02db-41c6-8fd0-3f3fee7ebace" (UID: "8cf419f0-02db-41c6-8fd0-3f3fee7ebace"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:22:26.376295 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:26.376272 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-kube-api-access-4pmst" (OuterVolumeSpecName: "kube-api-access-4pmst") pod "8cf419f0-02db-41c6-8fd0-3f3fee7ebace" (UID: "8cf419f0-02db-41c6-8fd0-3f3fee7ebace"). InnerVolumeSpecName "kube-api-access-4pmst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:22:26.475157 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:26.475128 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-extensions-socket-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:22:26.475157 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:26.475153 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pmst\" (UniqueName: \"kubernetes.io/projected/8cf419f0-02db-41c6-8fd0-3f3fee7ebace-kube-api-access-4pmst\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:22:27.160040 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:27.160005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:27.267123 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:27.267092 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:27.291854 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:27.291828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:27.294572 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:22:27.294548 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-ss5x5\" is forbidden: User \"system:node:ip-10-0-136-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-99.ec2.internal' and this object" podUID="8cf419f0-02db-41c6-8fd0-3f3fee7ebace" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:27.299694 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:22:27.299670 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-ss5x5\" is forbidden: User \"system:node:ip-10-0-136-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-99.ec2.internal' and this object" podUID="8cf419f0-02db-41c6-8fd0-3f3fee7ebace" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-ss5x5" Aug 04 19:22:27.602891 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:27.602863 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cf419f0-02db-41c6-8fd0-3f3fee7ebace" path="/var/lib/kubelet/pods/8cf419f0-02db-41c6-8fd0-3f3fee7ebace/volumes" Aug 04 19:22:28.084594 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:28.084554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:28.184727 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:28.184693 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:30.696266 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:30.696220 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:30.795104 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:30.795074 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:33.636524 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:33.636488 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:33.735521 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:33.735486 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:38.437100 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:38.437067 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:38.537115 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:38.537075 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:38.643437 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:38.643406 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:39.444624 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:39.444589 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:39.548489 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:39.548440 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:47.257092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:47.257046 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:47.366305 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:47.366270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:54.035972 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:54.035936 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:54.143837 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:54.143810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:55.785850 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:55.785803 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:55.802479 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:55.802447 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5jdt9"] Aug 04 19:22:55.819235 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:55.819209 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5jdt9"] Aug 04 19:22:55.819372 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:55.819327 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:22:55.822159 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:55.822133 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 19:22:55.899315 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:55.899285 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:55.900441 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:55.900422 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5jdt9"] Aug 04 19:22:55.906393 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:55.906374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1efbb9d9-8a8b-4208-8037-6f06295a93db-config-file\") pod \"limitador-limitador-859c4d96d6-5jdt9\" (UID: \"1efbb9d9-8a8b-4208-8037-6f06295a93db\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:22:55.906451 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:55.906405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl2s4\" (UniqueName: \"kubernetes.io/projected/1efbb9d9-8a8b-4208-8037-6f06295a93db-kube-api-access-pl2s4\") pod \"limitador-limitador-859c4d96d6-5jdt9\" (UID: \"1efbb9d9-8a8b-4208-8037-6f06295a93db\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:22:55.993121 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:55.993088 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:56.007284 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.007258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1efbb9d9-8a8b-4208-8037-6f06295a93db-config-file\") pod \"limitador-limitador-859c4d96d6-5jdt9\" (UID: \"1efbb9d9-8a8b-4208-8037-6f06295a93db\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:22:56.007369 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.007289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl2s4\" (UniqueName: \"kubernetes.io/projected/1efbb9d9-8a8b-4208-8037-6f06295a93db-kube-api-access-pl2s4\") pod \"limitador-limitador-859c4d96d6-5jdt9\" (UID: \"1efbb9d9-8a8b-4208-8037-6f06295a93db\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:22:56.007916 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.007900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1efbb9d9-8a8b-4208-8037-6f06295a93db-config-file\") pod \"limitador-limitador-859c4d96d6-5jdt9\" (UID: \"1efbb9d9-8a8b-4208-8037-6f06295a93db\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:22:56.018792 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.018768 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl2s4\" (UniqueName: \"kubernetes.io/projected/1efbb9d9-8a8b-4208-8037-6f06295a93db-kube-api-access-pl2s4\") pod \"limitador-limitador-859c4d96d6-5jdt9\" (UID: \"1efbb9d9-8a8b-4208-8037-6f06295a93db\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:22:56.130709 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.130660 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:22:56.257497 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.257434 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5jdt9"] Aug 04 19:22:56.259702 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:22:56.259673 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efbb9d9_8a8b_4208_8037_6f06295a93db.slice/crio-ea2ca08edcb4d85c898153b948c66d193c1de566e98b2384dbed9d986e39f156 WatchSource:0}: Error finding container ea2ca08edcb4d85c898153b948c66d193c1de566e98b2384dbed9d986e39f156: Status 404 returned error can't find the container with id ea2ca08edcb4d85c898153b948c66d193c1de566e98b2384dbed9d986e39f156 Aug 04 19:22:56.416013 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.415935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" event={"ID":"1efbb9d9-8a8b-4208-8037-6f06295a93db","Type":"ContainerStarted","Data":"ea2ca08edcb4d85c898153b948c66d193c1de566e98b2384dbed9d986e39f156"} Aug 04 19:22:56.610405 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.610367 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-2bv2w"] Aug 04 19:22:56.614823 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.614801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2bv2w" Aug 04 19:22:56.617670 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.617650 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dxk86\"" Aug 04 19:22:56.626913 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.626893 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2bv2w"] Aug 04 19:22:56.658093 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.658060 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:56.712140 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.712118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxgr4\" (UniqueName: \"kubernetes.io/projected/36c14c95-a972-463f-b046-cb085cb124b2-kube-api-access-nxgr4\") pod \"authorino-b54df94b4-2bv2w\" (UID: \"36c14c95-a972-463f-b046-cb085cb124b2\") " pod="kuadrant-system/authorino-b54df94b4-2bv2w" Aug 04 19:22:56.764013 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.763983 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:22:56.781632 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.781610 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8c25j"] Aug 04 19:22:56.784738 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.784723 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8c25j" Aug 04 19:22:56.797290 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.797269 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8c25j"] Aug 04 19:22:56.812578 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.812557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxgr4\" (UniqueName: \"kubernetes.io/projected/36c14c95-a972-463f-b046-cb085cb124b2-kube-api-access-nxgr4\") pod \"authorino-b54df94b4-2bv2w\" (UID: \"36c14c95-a972-463f-b046-cb085cb124b2\") " pod="kuadrant-system/authorino-b54df94b4-2bv2w" Aug 04 19:22:56.821102 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.821084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxgr4\" (UniqueName: \"kubernetes.io/projected/36c14c95-a972-463f-b046-cb085cb124b2-kube-api-access-nxgr4\") pod \"authorino-b54df94b4-2bv2w\" (UID: \"36c14c95-a972-463f-b046-cb085cb124b2\") " pod="kuadrant-system/authorino-b54df94b4-2bv2w" Aug 04 19:22:56.912997 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.912974 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxwt\" (UniqueName: \"kubernetes.io/projected/7861516c-1036-4aa2-a4d3-470fdb12a64a-kube-api-access-cxxwt\") pod \"authorino-c548b5d7d-8c25j\" (UID: \"7861516c-1036-4aa2-a4d3-470fdb12a64a\") " pod="kuadrant-system/authorino-c548b5d7d-8c25j" Aug 04 19:22:56.923917 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:56.923899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2bv2w" Aug 04 19:22:57.014026 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:57.013995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxwt\" (UniqueName: \"kubernetes.io/projected/7861516c-1036-4aa2-a4d3-470fdb12a64a-kube-api-access-cxxwt\") pod \"authorino-c548b5d7d-8c25j\" (UID: \"7861516c-1036-4aa2-a4d3-470fdb12a64a\") " pod="kuadrant-system/authorino-c548b5d7d-8c25j" Aug 04 19:22:57.024344 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:57.024315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxwt\" (UniqueName: \"kubernetes.io/projected/7861516c-1036-4aa2-a4d3-470fdb12a64a-kube-api-access-cxxwt\") pod \"authorino-c548b5d7d-8c25j\" (UID: \"7861516c-1036-4aa2-a4d3-470fdb12a64a\") " pod="kuadrant-system/authorino-c548b5d7d-8c25j" Aug 04 19:22:57.049104 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:57.049077 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2bv2w"] Aug 04 19:22:57.051804 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:22:57.051779 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36c14c95_a972_463f_b046_cb085cb124b2.slice/crio-a8444a4b3b6bead410182d486ea7c92d87c5c1a03ae302e7ee669d0a6a3b13d3 WatchSource:0}: Error finding container a8444a4b3b6bead410182d486ea7c92d87c5c1a03ae302e7ee669d0a6a3b13d3: Status 404 returned error can't find the container with id a8444a4b3b6bead410182d486ea7c92d87c5c1a03ae302e7ee669d0a6a3b13d3 Aug 04 19:22:57.093718 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:57.093696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8c25j" Aug 04 19:22:57.219884 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:57.219860 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8c25j"] Aug 04 19:22:57.222684 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:22:57.222619 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7861516c_1036_4aa2_a4d3_470fdb12a64a.slice/crio-8b0aba81a660cb5244aece249e4cd57d324c06697ae80a4ff7dd5f2c7dfed190 WatchSource:0}: Error finding container 8b0aba81a660cb5244aece249e4cd57d324c06697ae80a4ff7dd5f2c7dfed190: Status 404 returned error can't find the container with id 8b0aba81a660cb5244aece249e4cd57d324c06697ae80a4ff7dd5f2c7dfed190 Aug 04 19:22:57.422528 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:57.422488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2bv2w" event={"ID":"36c14c95-a972-463f-b046-cb085cb124b2","Type":"ContainerStarted","Data":"a8444a4b3b6bead410182d486ea7c92d87c5c1a03ae302e7ee669d0a6a3b13d3"} Aug 04 19:22:57.423829 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:22:57.423803 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8c25j" event={"ID":"7861516c-1036-4aa2-a4d3-470fdb12a64a","Type":"ContainerStarted","Data":"8b0aba81a660cb5244aece249e4cd57d324c06697ae80a4ff7dd5f2c7dfed190"} Aug 04 19:23:00.439148 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:00.439107 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" event={"ID":"1efbb9d9-8a8b-4208-8037-6f06295a93db","Type":"ContainerStarted","Data":"4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8"} Aug 04 19:23:00.439590 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:00.439252 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:23:00.440592 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:00.440571 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2bv2w" event={"ID":"36c14c95-a972-463f-b046-cb085cb124b2","Type":"ContainerStarted","Data":"7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2"} Aug 04 19:23:00.441979 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:00.441959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8c25j" event={"ID":"7861516c-1036-4aa2-a4d3-470fdb12a64a","Type":"ContainerStarted","Data":"1e85e80b8f2c62f6c62c3c4ca91a041d7cb5d252e87740fc540913ed09d74369"} Aug 04 19:23:00.456729 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:00.456675 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" podStartSLOduration=2.712834625 podStartE2EDuration="5.456660354s" totalImagesPullingTime="2.743825729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:23:00.456053731 +0000 UTC m=+425.361654295" watchObservedRunningTime="2026-08-04 19:23:00.456660354 +0000 UTC m=+425.362260894" Aug 04 19:23:00.470539 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:00.470490 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-8c25j" podStartSLOduration=1.7723362009999999 podStartE2EDuration="4.470443336s" totalImagesPullingTime="2.698107135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:23:00.470363358 +0000 UTC m=+425.375963898" watchObservedRunningTime="2026-08-04 19:23:00.470443336 +0000 UTC m=+425.376043875" Aug 04 19:23:00.506261 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:00.506201 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-2bv2w" podStartSLOduration=1.94251492 podStartE2EDuration="4.506181368s" totalImagesPullingTime="2.563666448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:23:00.484407577 +0000 UTC m=+425.390008115" watchObservedRunningTime="2026-08-04 19:23:00.506181368 +0000 UTC m=+425.411781906" Aug 04 19:23:00.507003 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:00.506984 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2bv2w"] Aug 04 19:23:02.450170 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:02.450129 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-2bv2w" podUID="36c14c95-a972-463f-b046-cb085cb124b2" containerName="authorino" containerID="cri-o://7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2" gracePeriod=30 Aug 04 19:23:02.694502 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:02.694459 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2bv2w" Aug 04 19:23:02.864457 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:02.864374 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxgr4\" (UniqueName: \"kubernetes.io/projected/36c14c95-a972-463f-b046-cb085cb124b2-kube-api-access-nxgr4\") pod \"36c14c95-a972-463f-b046-cb085cb124b2\" (UID: \"36c14c95-a972-463f-b046-cb085cb124b2\") " Aug 04 19:23:02.866647 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:02.866625 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c14c95-a972-463f-b046-cb085cb124b2-kube-api-access-nxgr4" (OuterVolumeSpecName: "kube-api-access-nxgr4") pod "36c14c95-a972-463f-b046-cb085cb124b2" (UID: "36c14c95-a972-463f-b046-cb085cb124b2"). InnerVolumeSpecName "kube-api-access-nxgr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:23:02.965203 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:02.965169 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxgr4\" (UniqueName: \"kubernetes.io/projected/36c14c95-a972-463f-b046-cb085cb124b2-kube-api-access-nxgr4\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:23:03.455290 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:03.455259 2580 generic.go:356] "Generic (PLEG): container finished" podID="36c14c95-a972-463f-b046-cb085cb124b2" containerID="7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2" exitCode=0 Aug 04 19:23:03.455746 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:03.455297 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2bv2w" Aug 04 19:23:03.455746 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:03.455350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2bv2w" event={"ID":"36c14c95-a972-463f-b046-cb085cb124b2","Type":"ContainerDied","Data":"7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2"} Aug 04 19:23:03.455746 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:03.455381 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2bv2w" event={"ID":"36c14c95-a972-463f-b046-cb085cb124b2","Type":"ContainerDied","Data":"a8444a4b3b6bead410182d486ea7c92d87c5c1a03ae302e7ee669d0a6a3b13d3"} Aug 04 19:23:03.455746 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:03.455396 2580 scope.go:117] "RemoveContainer" containerID="7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2" Aug 04 19:23:03.464161 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:03.464132 2580 scope.go:117] "RemoveContainer" containerID="7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2" Aug 04 19:23:03.464410 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:23:03.464390 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2\": container with ID starting with 7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2 not found: ID does not exist" containerID="7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2" Aug 04 19:23:03.464483 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:03.464418 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2"} err="failed to get container status \"7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2\": rpc error: code = NotFound desc = could not find container \"7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2\": container with ID starting with 7506ac673eab0df3afb492ddefdb5ebb51232feb36ac970c4685daf8628506c2 not found: ID does not exist" Aug 04 19:23:03.477020 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:03.476998 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2bv2w"] Aug 04 19:23:03.482388 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:03.482362 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2bv2w"] Aug 04 19:23:03.602814 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:03.602786 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36c14c95-a972-463f-b046-cb085cb124b2" path="/var/lib/kubelet/pods/36c14c95-a972-463f-b046-cb085cb124b2/volumes" Aug 04 19:23:09.661536 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:09.661507 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5jdt9"] Aug 04 19:23:09.661924 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:09.661697 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" podUID="1efbb9d9-8a8b-4208-8037-6f06295a93db" containerName="limitador" containerID="cri-o://4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8" gracePeriod=30 Aug 04 19:23:09.662502 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:09.662341 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:23:10.216959 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.216938 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:23:10.322277 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.322209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1efbb9d9-8a8b-4208-8037-6f06295a93db-config-file\") pod \"1efbb9d9-8a8b-4208-8037-6f06295a93db\" (UID: \"1efbb9d9-8a8b-4208-8037-6f06295a93db\") " Aug 04 19:23:10.322277 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.322258 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl2s4\" (UniqueName: \"kubernetes.io/projected/1efbb9d9-8a8b-4208-8037-6f06295a93db-kube-api-access-pl2s4\") pod \"1efbb9d9-8a8b-4208-8037-6f06295a93db\" (UID: \"1efbb9d9-8a8b-4208-8037-6f06295a93db\") " Aug 04 19:23:10.322593 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.322571 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efbb9d9-8a8b-4208-8037-6f06295a93db-config-file" (OuterVolumeSpecName: "config-file") pod "1efbb9d9-8a8b-4208-8037-6f06295a93db" (UID: "1efbb9d9-8a8b-4208-8037-6f06295a93db"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:23:10.324373 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.324343 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efbb9d9-8a8b-4208-8037-6f06295a93db-kube-api-access-pl2s4" (OuterVolumeSpecName: "kube-api-access-pl2s4") pod "1efbb9d9-8a8b-4208-8037-6f06295a93db" (UID: "1efbb9d9-8a8b-4208-8037-6f06295a93db"). InnerVolumeSpecName "kube-api-access-pl2s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:23:10.423701 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.423667 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1efbb9d9-8a8b-4208-8037-6f06295a93db-config-file\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:23:10.423701 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.423695 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl2s4\" (UniqueName: \"kubernetes.io/projected/1efbb9d9-8a8b-4208-8037-6f06295a93db-kube-api-access-pl2s4\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:23:10.488169 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.488136 2580 generic.go:356] "Generic (PLEG): container finished" podID="1efbb9d9-8a8b-4208-8037-6f06295a93db" containerID="4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8" exitCode=0 Aug 04 19:23:10.488315 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.488169 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" event={"ID":"1efbb9d9-8a8b-4208-8037-6f06295a93db","Type":"ContainerDied","Data":"4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8"} Aug 04 19:23:10.488315 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.488190 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" Aug 04 19:23:10.488315 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.488222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-5jdt9" event={"ID":"1efbb9d9-8a8b-4208-8037-6f06295a93db","Type":"ContainerDied","Data":"ea2ca08edcb4d85c898153b948c66d193c1de566e98b2384dbed9d986e39f156"} Aug 04 19:23:10.488315 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.488245 2580 scope.go:117] "RemoveContainer" containerID="4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8" Aug 04 19:23:10.497131 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.497115 2580 scope.go:117] "RemoveContainer" containerID="4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8" Aug 04 19:23:10.497373 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:23:10.497356 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8\": container with ID starting with 4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8 not found: ID does not exist" containerID="4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8" Aug 04 19:23:10.497421 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.497383 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8"} err="failed to get container status \"4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8\": rpc error: code = NotFound desc = could not find container \"4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8\": container with ID starting with 4c74d1d9f22ff19c2c00758e81a8f8034ac8e32eec6a9bb31d201c34f4ba57d8 not found: ID does not exist" Aug 04 19:23:10.508417 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.508398 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5jdt9"] Aug 04 19:23:10.512561 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:10.512543 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5jdt9"] Aug 04 19:23:11.603096 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:11.603067 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1efbb9d9-8a8b-4208-8037-6f06295a93db" path="/var/lib/kubelet/pods/1efbb9d9-8a8b-4208-8037-6f06295a93db/volumes" Aug 04 19:23:11.817427 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:11.817398 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:11.917295 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:11.917212 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:12.239822 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.239793 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-f28jh"] Aug 04 19:23:12.240102 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.240090 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c14c95-a972-463f-b046-cb085cb124b2" containerName="authorino" Aug 04 19:23:12.240141 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.240104 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c14c95-a972-463f-b046-cb085cb124b2" containerName="authorino" Aug 04 19:23:12.240141 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.240124 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1efbb9d9-8a8b-4208-8037-6f06295a93db" containerName="limitador" Aug 04 19:23:12.240141 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.240130 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1efbb9d9-8a8b-4208-8037-6f06295a93db" containerName="limitador" Aug 04 19:23:12.240234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.240177 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1efbb9d9-8a8b-4208-8037-6f06295a93db" containerName="limitador" Aug 04 19:23:12.240234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.240188 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="36c14c95-a972-463f-b046-cb085cb124b2" containerName="authorino" Aug 04 19:23:12.246742 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.246718 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" Aug 04 19:23:12.249864 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.249832 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 19:23:12.249864 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.249855 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 19:23:12.251269 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.250905 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Aug 04 19:23:12.251269 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.250905 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-mcj4j\"" Aug 04 19:23:12.253164 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.253140 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-f28jh"] Aug 04 19:23:12.300992 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.300965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:12.336723 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.336698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7dv\" (UniqueName: \"kubernetes.io/projected/2f301fa9-6a66-47fb-a7f8-41f8ec6014f2-kube-api-access-vh7dv\") pod \"postgres-69fb4594cf-f28jh\" (UID: \"2f301fa9-6a66-47fb-a7f8-41f8ec6014f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" Aug 04 19:23:12.336844 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.336745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2f301fa9-6a66-47fb-a7f8-41f8ec6014f2-data\") pod \"postgres-69fb4594cf-f28jh\" (UID: \"2f301fa9-6a66-47fb-a7f8-41f8ec6014f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" Aug 04 19:23:12.437636 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.437599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7dv\" (UniqueName: \"kubernetes.io/projected/2f301fa9-6a66-47fb-a7f8-41f8ec6014f2-kube-api-access-vh7dv\") pod \"postgres-69fb4594cf-f28jh\" (UID: \"2f301fa9-6a66-47fb-a7f8-41f8ec6014f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" Aug 04 19:23:12.437775 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.437657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2f301fa9-6a66-47fb-a7f8-41f8ec6014f2-data\") pod \"postgres-69fb4594cf-f28jh\" (UID: \"2f301fa9-6a66-47fb-a7f8-41f8ec6014f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" Aug 04 19:23:12.438018 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.438001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2f301fa9-6a66-47fb-a7f8-41f8ec6014f2-data\") pod \"postgres-69fb4594cf-f28jh\" (UID: \"2f301fa9-6a66-47fb-a7f8-41f8ec6014f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" Aug 04 19:23:12.446706 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.446682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7dv\" (UniqueName: \"kubernetes.io/projected/2f301fa9-6a66-47fb-a7f8-41f8ec6014f2-kube-api-access-vh7dv\") pod \"postgres-69fb4594cf-f28jh\" (UID: \"2f301fa9-6a66-47fb-a7f8-41f8ec6014f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" Aug 04 19:23:12.557619 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.557552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" Aug 04 19:23:12.688602 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:12.688578 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-f28jh"] Aug 04 19:23:12.690550 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:23:12.690522 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f301fa9_6a66_47fb_a7f8_41f8ec6014f2.slice/crio-443a34c124587d2d6d0529984dedd0f3280ff19706d3e840ac140a7f19422cb3 WatchSource:0}: Error finding container 443a34c124587d2d6d0529984dedd0f3280ff19706d3e840ac140a7f19422cb3: Status 404 returned error can't find the container with id 443a34c124587d2d6d0529984dedd0f3280ff19706d3e840ac140a7f19422cb3 Aug 04 19:23:13.504124 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:13.504092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" event={"ID":"2f301fa9-6a66-47fb-a7f8-41f8ec6014f2","Type":"ContainerStarted","Data":"443a34c124587d2d6d0529984dedd0f3280ff19706d3e840ac140a7f19422cb3"} Aug 04 19:23:18.529561 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:18.529527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" event={"ID":"2f301fa9-6a66-47fb-a7f8-41f8ec6014f2","Type":"ContainerStarted","Data":"1bdee8b984bf1dca371fc813b46633cbf28ca2d6858c1339420d9a07ba2fa1bd"} Aug 04 19:23:18.529984 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:18.529675 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" Aug 04 19:23:18.545020 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:18.544982 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" podStartSLOduration=1.575799812 podStartE2EDuration="6.544969758s" totalImagesPullingTime="4.969169946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:23:18.544723137 +0000 UTC m=+443.450323743" watchObservedRunningTime="2026-08-04 19:23:18.544969758 +0000 UTC m=+443.450570296" Aug 04 19:23:24.562181 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:24.562149 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-f28jh" Aug 04 19:23:24.899681 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:24.899604 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:25.113520 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.113489 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ps56t"] Aug 04 19:23:25.117852 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.117831 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ps56t" Aug 04 19:23:25.124683 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.124522 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ps56t"] Aug 04 19:23:25.244891 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.244857 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:25.245102 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.245082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxcqf\" (UniqueName: \"kubernetes.io/projected/0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf-kube-api-access-gxcqf\") pod \"authorino-5dc7ccb5f-ps56t\" (UID: \"0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf\") " pod="kuadrant-system/authorino-5dc7ccb5f-ps56t" Aug 04 19:23:25.334956 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.334927 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ps56t"] Aug 04 19:23:25.335117 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:23:25.335098 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gxcqf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-ps56t" podUID="0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf" Aug 04 19:23:25.345840 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.345817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxcqf\" (UniqueName: \"kubernetes.io/projected/0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf-kube-api-access-gxcqf\") pod \"authorino-5dc7ccb5f-ps56t\" (UID: \"0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf\") " pod="kuadrant-system/authorino-5dc7ccb5f-ps56t" Aug 04 19:23:25.358664 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.358642 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxcqf\" (UniqueName: \"kubernetes.io/projected/0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf-kube-api-access-gxcqf\") pod \"authorino-5dc7ccb5f-ps56t\" (UID: \"0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf\") " pod="kuadrant-system/authorino-5dc7ccb5f-ps56t" Aug 04 19:23:25.361988 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.361971 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bd48f69c6-qmw97"] Aug 04 19:23:25.364151 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.364137 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd48f69c6-qmw97" Aug 04 19:23:25.377044 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.377023 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bd48f69c6-qmw97"] Aug 04 19:23:25.437782 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.437750 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bd48f69c6-qmw97"] Aug 04 19:23:25.438016 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:23:25.437994 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gj95t], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7bd48f69c6-qmw97" podUID="1956a767-e8b0-4c2a-8169-594fd1154c45" Aug 04 19:23:25.446235 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.446208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj95t\" (UniqueName: \"kubernetes.io/projected/1956a767-e8b0-4c2a-8169-594fd1154c45-kube-api-access-gj95t\") pod \"authorino-7bd48f69c6-qmw97\" (UID: \"1956a767-e8b0-4c2a-8169-594fd1154c45\") " pod="kuadrant-system/authorino-7bd48f69c6-qmw97" Aug 04 19:23:25.547617 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.547539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj95t\" (UniqueName: \"kubernetes.io/projected/1956a767-e8b0-4c2a-8169-594fd1154c45-kube-api-access-gj95t\") pod \"authorino-7bd48f69c6-qmw97\" (UID: \"1956a767-e8b0-4c2a-8169-594fd1154c45\") " pod="kuadrant-system/authorino-7bd48f69c6-qmw97" Aug 04 19:23:25.557002 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.556978 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj95t\" (UniqueName: \"kubernetes.io/projected/1956a767-e8b0-4c2a-8169-594fd1154c45-kube-api-access-gj95t\") pod \"authorino-7bd48f69c6-qmw97\" (UID: \"1956a767-e8b0-4c2a-8169-594fd1154c45\") " pod="kuadrant-system/authorino-7bd48f69c6-qmw97" Aug 04 19:23:25.558269 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.558251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ps56t" Aug 04 19:23:25.558357 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.558251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd48f69c6-qmw97" Aug 04 19:23:25.569479 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.569451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ps56t" Aug 04 19:23:25.571786 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.571767 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd48f69c6-qmw97" Aug 04 19:23:25.648104 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.648079 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxcqf\" (UniqueName: \"kubernetes.io/projected/0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf-kube-api-access-gxcqf\") pod \"0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf\" (UID: \"0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf\") " Aug 04 19:23:25.648227 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.648120 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj95t\" (UniqueName: \"kubernetes.io/projected/1956a767-e8b0-4c2a-8169-594fd1154c45-kube-api-access-gj95t\") pod \"1956a767-e8b0-4c2a-8169-594fd1154c45\" (UID: \"1956a767-e8b0-4c2a-8169-594fd1154c45\") " Aug 04 19:23:25.650273 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.650249 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1956a767-e8b0-4c2a-8169-594fd1154c45-kube-api-access-gj95t" (OuterVolumeSpecName: "kube-api-access-gj95t") pod "1956a767-e8b0-4c2a-8169-594fd1154c45" (UID: "1956a767-e8b0-4c2a-8169-594fd1154c45"). InnerVolumeSpecName "kube-api-access-gj95t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:23:25.650371 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.650297 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf-kube-api-access-gxcqf" (OuterVolumeSpecName: "kube-api-access-gxcqf") pod "0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf" (UID: "0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf"). InnerVolumeSpecName "kube-api-access-gxcqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:23:25.749070 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.749043 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxcqf\" (UniqueName: \"kubernetes.io/projected/0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf-kube-api-access-gxcqf\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:23:25.749070 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:25.749066 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj95t\" (UniqueName: \"kubernetes.io/projected/1956a767-e8b0-4c2a-8169-594fd1154c45-kube-api-access-gj95t\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:23:26.562014 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:26.561985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ps56t" Aug 04 19:23:26.562180 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:26.561985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd48f69c6-qmw97" Aug 04 19:23:26.594238 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:26.594210 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ps56t"] Aug 04 19:23:26.597066 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:26.597044 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ps56t"] Aug 04 19:23:26.620672 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:26.620650 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bd48f69c6-qmw97"] Aug 04 19:23:26.625208 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:26.625187 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bd48f69c6-qmw97"] Aug 04 19:23:27.602232 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:27.602195 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf" path="/var/lib/kubelet/pods/0db7c9c9-d0d0-45e5-b395-81b9ff4adbcf/volumes" Aug 04 19:23:27.602617 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:27.602386 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1956a767-e8b0-4c2a-8169-594fd1154c45" path="/var/lib/kubelet/pods/1956a767-e8b0-4c2a-8169-594fd1154c45/volumes" Aug 04 19:23:28.915331 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:28.915303 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8c25j"] Aug 04 19:23:28.915766 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:28.915481 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-8c25j" podUID="7861516c-1036-4aa2-a4d3-470fdb12a64a" containerName="authorino" containerID="cri-o://1e85e80b8f2c62f6c62c3c4ca91a041d7cb5d252e87740fc540913ed09d74369" gracePeriod=30 Aug 04 19:23:29.581371 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:29.581339 2580 generic.go:356] "Generic (PLEG): container finished" podID="7861516c-1036-4aa2-a4d3-470fdb12a64a" containerID="1e85e80b8f2c62f6c62c3c4ca91a041d7cb5d252e87740fc540913ed09d74369" exitCode=0 Aug 04 19:23:29.581615 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:29.581410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8c25j" event={"ID":"7861516c-1036-4aa2-a4d3-470fdb12a64a","Type":"ContainerDied","Data":"1e85e80b8f2c62f6c62c3c4ca91a041d7cb5d252e87740fc540913ed09d74369"} Aug 04 19:23:29.700813 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:29.700785 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8c25j" Aug 04 19:23:29.782710 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:29.782682 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxxwt\" (UniqueName: \"kubernetes.io/projected/7861516c-1036-4aa2-a4d3-470fdb12a64a-kube-api-access-cxxwt\") pod \"7861516c-1036-4aa2-a4d3-470fdb12a64a\" (UID: \"7861516c-1036-4aa2-a4d3-470fdb12a64a\") " Aug 04 19:23:29.784780 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:29.784753 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7861516c-1036-4aa2-a4d3-470fdb12a64a-kube-api-access-cxxwt" (OuterVolumeSpecName: "kube-api-access-cxxwt") pod "7861516c-1036-4aa2-a4d3-470fdb12a64a" (UID: "7861516c-1036-4aa2-a4d3-470fdb12a64a"). InnerVolumeSpecName "kube-api-access-cxxwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:23:29.883877 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:29.883845 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxxwt\" (UniqueName: \"kubernetes.io/projected/7861516c-1036-4aa2-a4d3-470fdb12a64a-kube-api-access-cxxwt\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:23:30.587199 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:30.587166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8c25j" event={"ID":"7861516c-1036-4aa2-a4d3-470fdb12a64a","Type":"ContainerDied","Data":"8b0aba81a660cb5244aece249e4cd57d324c06697ae80a4ff7dd5f2c7dfed190"} Aug 04 19:23:30.587199 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:30.587192 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8c25j" Aug 04 19:23:30.587690 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:30.587219 2580 scope.go:117] "RemoveContainer" containerID="1e85e80b8f2c62f6c62c3c4ca91a041d7cb5d252e87740fc540913ed09d74369" Aug 04 19:23:30.611089 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:30.611064 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8c25j"] Aug 04 19:23:30.617670 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:30.617648 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8c25j"] Aug 04 19:23:31.602680 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:31.602647 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7861516c-1036-4aa2-a4d3-470fdb12a64a" path="/var/lib/kubelet/pods/7861516c-1036-4aa2-a4d3-470fdb12a64a/volumes" Aug 04 19:23:33.693366 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.693338 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:33.708067 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.708036 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-794485cd76-tgr7c"] Aug 04 19:23:33.708377 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.708366 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7861516c-1036-4aa2-a4d3-470fdb12a64a" containerName="authorino" Aug 04 19:23:33.708420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.708379 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7861516c-1036-4aa2-a4d3-470fdb12a64a" containerName="authorino" Aug 04 19:23:33.708453 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.708427 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7861516c-1036-4aa2-a4d3-470fdb12a64a" containerName="authorino" Aug 04 19:23:33.713765 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.713748 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:33.716429 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.716409 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 19:23:33.716561 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.716409 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 19:23:33.716755 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.716743 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vp2lx\"" Aug 04 19:23:33.726385 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.726365 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-794485cd76-tgr7c"] Aug 04 19:23:33.778291 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.778264 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:33.813018 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.812995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28ftj\" (UniqueName: \"kubernetes.io/projected/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-kube-api-access-28ftj\") pod \"maas-controller-794485cd76-tgr7c\" (UID: \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\") " pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:33.813110 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.813045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-webhook-cert\") pod \"maas-controller-794485cd76-tgr7c\" (UID: \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\") " pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:33.850806 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.850783 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:33.877664 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.877637 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bbff758-5pfkj"] Aug 04 19:23:33.880069 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.880055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:33.893344 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.893320 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbff758-5pfkj"] Aug 04 19:23:33.913443 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.913423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-webhook-cert\") pod \"maas-controller-794485cd76-tgr7c\" (UID: \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\") " pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:33.913569 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.913502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28ftj\" (UniqueName: \"kubernetes.io/projected/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-kube-api-access-28ftj\") pod \"maas-controller-794485cd76-tgr7c\" (UID: \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\") " pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:33.913613 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:23:33.913567 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Aug 04 19:23:33.913653 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:23:33.913638 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-webhook-cert podName:d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8 nodeName:}" failed. No retries permitted until 2026-08-04 19:23:34.413620731 +0000 UTC m=+459.319221248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-webhook-cert") pod "maas-controller-794485cd76-tgr7c" (UID: "d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8") : secret "maas-controller-webhook-cert" not found Aug 04 19:23:33.924196 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:33.924168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28ftj\" (UniqueName: \"kubernetes.io/projected/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-kube-api-access-28ftj\") pod \"maas-controller-794485cd76-tgr7c\" (UID: \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\") " pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:34.014443 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.014373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms556\" (UniqueName: \"kubernetes.io/projected/26b7a1c3-1ac7-4738-b106-e292d29b1213-kube-api-access-ms556\") pod \"maas-controller-7bbff758-5pfkj\" (UID: \"26b7a1c3-1ac7-4738-b106-e292d29b1213\") " pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:34.014443 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.014415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b7a1c3-1ac7-4738-b106-e292d29b1213-webhook-cert\") pod \"maas-controller-7bbff758-5pfkj\" (UID: \"26b7a1c3-1ac7-4738-b106-e292d29b1213\") " pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:34.115319 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.115287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms556\" (UniqueName: \"kubernetes.io/projected/26b7a1c3-1ac7-4738-b106-e292d29b1213-kube-api-access-ms556\") pod \"maas-controller-7bbff758-5pfkj\" (UID: \"26b7a1c3-1ac7-4738-b106-e292d29b1213\") " pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:34.115451 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.115340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b7a1c3-1ac7-4738-b106-e292d29b1213-webhook-cert\") pod \"maas-controller-7bbff758-5pfkj\" (UID: \"26b7a1c3-1ac7-4738-b106-e292d29b1213\") " pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:34.115524 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:23:34.115513 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Aug 04 19:23:34.115590 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:23:34.115576 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b7a1c3-1ac7-4738-b106-e292d29b1213-webhook-cert podName:26b7a1c3-1ac7-4738-b106-e292d29b1213 nodeName:}" failed. No retries permitted until 2026-08-04 19:23:34.615561241 +0000 UTC m=+459.521161760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/26b7a1c3-1ac7-4738-b106-e292d29b1213-webhook-cert") pod "maas-controller-7bbff758-5pfkj" (UID: "26b7a1c3-1ac7-4738-b106-e292d29b1213") : secret "maas-controller-webhook-cert" not found Aug 04 19:23:34.124064 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.124047 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms556\" (UniqueName: \"kubernetes.io/projected/26b7a1c3-1ac7-4738-b106-e292d29b1213-kube-api-access-ms556\") pod \"maas-controller-7bbff758-5pfkj\" (UID: \"26b7a1c3-1ac7-4738-b106-e292d29b1213\") " pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:34.252505 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.252445 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:34.418315 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.418238 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-webhook-cert\") pod \"maas-controller-794485cd76-tgr7c\" (UID: \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\") " pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:34.420744 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.420728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-webhook-cert\") pod \"maas-controller-794485cd76-tgr7c\" (UID: \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\") " pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:34.619994 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.619965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b7a1c3-1ac7-4738-b106-e292d29b1213-webhook-cert\") pod \"maas-controller-7bbff758-5pfkj\" (UID: \"26b7a1c3-1ac7-4738-b106-e292d29b1213\") " pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:34.622273 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.622255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b7a1c3-1ac7-4738-b106-e292d29b1213-webhook-cert\") pod \"maas-controller-7bbff758-5pfkj\" (UID: \"26b7a1c3-1ac7-4738-b106-e292d29b1213\") " pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:34.624149 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.624135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:34.749555 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.749531 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-794485cd76-tgr7c"] Aug 04 19:23:34.751758 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:23:34.751734 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d8a4fd_19b4_4bb8_9355_fdb48cb0f3d8.slice/crio-976684e0c3648a65d86c2649122b30abfbd1aac6206e29e0df209310f5471ea2 WatchSource:0}: Error finding container 976684e0c3648a65d86c2649122b30abfbd1aac6206e29e0df209310f5471ea2: Status 404 returned error can't find the container with id 976684e0c3648a65d86c2649122b30abfbd1aac6206e29e0df209310f5471ea2 Aug 04 19:23:34.789723 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.789697 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:34.914719 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:34.914695 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbff758-5pfkj"] Aug 04 19:23:34.917566 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:23:34.917534 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b7a1c3_1ac7_4738_b106_e292d29b1213.slice/crio-df9bb13d5fc0f640b02a4bde2074ee140476dfa4d60234e407a3f1d17eef846f WatchSource:0}: Error finding container df9bb13d5fc0f640b02a4bde2074ee140476dfa4d60234e407a3f1d17eef846f: Status 404 returned error can't find the container with id df9bb13d5fc0f640b02a4bde2074ee140476dfa4d60234e407a3f1d17eef846f Aug 04 19:23:35.611207 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:35.611162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794485cd76-tgr7c" event={"ID":"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8","Type":"ContainerStarted","Data":"976684e0c3648a65d86c2649122b30abfbd1aac6206e29e0df209310f5471ea2"} Aug 04 19:23:35.612454 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:35.612426 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbff758-5pfkj" event={"ID":"26b7a1c3-1ac7-4738-b106-e292d29b1213","Type":"ContainerStarted","Data":"df9bb13d5fc0f640b02a4bde2074ee140476dfa4d60234e407a3f1d17eef846f"} Aug 04 19:23:36.536814 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:36.536773 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:36.618384 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:36.618288 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbff758-5pfkj" event={"ID":"26b7a1c3-1ac7-4738-b106-e292d29b1213","Type":"ContainerStarted","Data":"a08033bb5b9f26226602c821100e3118e57902cf90ef0818ae1d0b80425c0433"} Aug 04 19:23:36.618593 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:36.618398 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:36.619636 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:36.619607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794485cd76-tgr7c" event={"ID":"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8","Type":"ContainerStarted","Data":"0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4"} Aug 04 19:23:36.619749 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:36.619731 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:36.640919 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:36.640892 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:23:36.643568 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:36.643527 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bbff758-5pfkj" podStartSLOduration=2.205099163 podStartE2EDuration="3.643513825s" totalImagesPullingTime="1.438414662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:23:36.643374884 +0000 UTC m=+461.548975423" watchObservedRunningTime="2026-08-04 19:23:36.643513825 +0000 UTC m=+461.549114364" Aug 04 19:23:36.665454 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:36.665414 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-794485cd76-tgr7c" podStartSLOduration=2.058687618 podStartE2EDuration="3.665401117s" totalImagesPullingTime="1.606713499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:23:36.664804351 +0000 UTC m=+461.570404889" watchObservedRunningTime="2026-08-04 19:23:36.665401117 +0000 UTC m=+461.571001655" Aug 04 19:23:47.625878 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:47.625836 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:47.626278 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:47.625921 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:23:47.702878 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:47.702847 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-794485cd76-tgr7c"] Aug 04 19:23:47.703099 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:47.703054 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-794485cd76-tgr7c" podUID="d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8" containerName="manager" containerID="cri-o://0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4" gracePeriod=30 Aug 04 19:23:47.941722 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:47.941703 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:48.031151 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.031126 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-webhook-cert\") pod \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\" (UID: \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\") " Aug 04 19:23:48.031269 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.031163 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28ftj\" (UniqueName: \"kubernetes.io/projected/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-kube-api-access-28ftj\") pod \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\" (UID: \"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8\") " Aug 04 19:23:48.033311 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.033277 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8" (UID: "d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:23:48.033414 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.033388 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-kube-api-access-28ftj" (OuterVolumeSpecName: "kube-api-access-28ftj") pod "d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8" (UID: "d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8"). InnerVolumeSpecName "kube-api-access-28ftj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:23:48.132049 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.132024 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-webhook-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:23:48.132049 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.132047 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28ftj\" (UniqueName: \"kubernetes.io/projected/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8-kube-api-access-28ftj\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:23:48.671789 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.671748 2580 generic.go:356] "Generic (PLEG): container finished" podID="d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8" containerID="0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4" exitCode=0 Aug 04 19:23:48.672221 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.671823 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794485cd76-tgr7c" Aug 04 19:23:48.672221 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.671838 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794485cd76-tgr7c" event={"ID":"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8","Type":"ContainerDied","Data":"0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4"} Aug 04 19:23:48.672221 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.671909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794485cd76-tgr7c" event={"ID":"d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8","Type":"ContainerDied","Data":"976684e0c3648a65d86c2649122b30abfbd1aac6206e29e0df209310f5471ea2"} Aug 04 19:23:48.672221 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.671945 2580 scope.go:117] "RemoveContainer" containerID="0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4" Aug 04 19:23:48.680612 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.680596 2580 scope.go:117] "RemoveContainer" containerID="0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4" Aug 04 19:23:48.680878 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:23:48.680859 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4\": container with ID starting with 0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4 not found: ID does not exist" containerID="0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4" Aug 04 19:23:48.680943 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.680887 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4"} err="failed to get container status \"0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4\": rpc error: code = NotFound desc = could not find container \"0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4\": container with ID starting with 0cc23716493af35646e9843e2bf937adfa220ad31690b1fbc139c57eefaf42d4 not found: ID does not exist" Aug 04 19:23:48.691925 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.691899 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-794485cd76-tgr7c"] Aug 04 19:23:48.698412 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:48.698392 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-794485cd76-tgr7c"] Aug 04 19:23:49.602689 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:23:49.602656 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8" path="/var/lib/kubelet/pods/d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8/volumes" Aug 04 19:24:05.294777 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:05.294737 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:05.390872 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:05.390841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:06.889566 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:06.889522 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:07.013154 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.013123 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67c8968587-5lb29"] Aug 04 19:24:07.013447 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.013436 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8" containerName="manager" Aug 04 19:24:07.013521 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.013450 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8" containerName="manager" Aug 04 19:24:07.013562 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.013546 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d8a4fd-19b4-4bb8-9355-fdb48cb0f3d8" containerName="manager" Aug 04 19:24:07.016391 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.016375 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.019010 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.018988 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Aug 04 19:24:07.019121 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.019065 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:24:07.024657 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.024634 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c8968587-5lb29"] Aug 04 19:24:07.086873 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.086842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ggk\" (UniqueName: \"kubernetes.io/projected/2c659b52-c5e9-483a-8311-d2ae90c856f7-kube-api-access-q9ggk\") pod \"maas-api-67c8968587-5lb29\" (UID: \"2c659b52-c5e9-483a-8311-d2ae90c856f7\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.087028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.086895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c659b52-c5e9-483a-8311-d2ae90c856f7-openshift-service-ca\") pod \"maas-api-67c8968587-5lb29\" (UID: \"2c659b52-c5e9-483a-8311-d2ae90c856f7\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.087028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.086983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c659b52-c5e9-483a-8311-d2ae90c856f7-maas-api-tls\") pod \"maas-api-67c8968587-5lb29\" (UID: \"2c659b52-c5e9-483a-8311-d2ae90c856f7\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.187630 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.187558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c659b52-c5e9-483a-8311-d2ae90c856f7-maas-api-tls\") pod \"maas-api-67c8968587-5lb29\" (UID: \"2c659b52-c5e9-483a-8311-d2ae90c856f7\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.187630 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.187603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ggk\" (UniqueName: \"kubernetes.io/projected/2c659b52-c5e9-483a-8311-d2ae90c856f7-kube-api-access-q9ggk\") pod \"maas-api-67c8968587-5lb29\" (UID: \"2c659b52-c5e9-483a-8311-d2ae90c856f7\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.187781 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.187644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c659b52-c5e9-483a-8311-d2ae90c856f7-openshift-service-ca\") pod \"maas-api-67c8968587-5lb29\" (UID: \"2c659b52-c5e9-483a-8311-d2ae90c856f7\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.188480 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.188389 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c659b52-c5e9-483a-8311-d2ae90c856f7-openshift-service-ca\") pod \"maas-api-67c8968587-5lb29\" (UID: \"2c659b52-c5e9-483a-8311-d2ae90c856f7\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.190182 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.190161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c659b52-c5e9-483a-8311-d2ae90c856f7-maas-api-tls\") pod \"maas-api-67c8968587-5lb29\" (UID: \"2c659b52-c5e9-483a-8311-d2ae90c856f7\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.198564 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.198543 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ggk\" (UniqueName: \"kubernetes.io/projected/2c659b52-c5e9-483a-8311-d2ae90c856f7-kube-api-access-q9ggk\") pod \"maas-api-67c8968587-5lb29\" (UID: \"2c659b52-c5e9-483a-8311-d2ae90c856f7\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.208510 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.208482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:07.327146 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.327115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:07.374448 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.374407 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:07.461647 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.461615 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x"] Aug 04 19:24:07.464658 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.464641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" Aug 04 19:24:07.464795 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.464776 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c8968587-5lb29"] Aug 04 19:24:07.467077 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.467057 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Aug 04 19:24:07.467531 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:24:07.467506 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c659b52_c5e9_483a_8311_d2ae90c856f7.slice/crio-1b27cdb8dd3d6e2506b7c877aa05a88aa04cfd17791fe4b841c72a3d908cadd8 WatchSource:0}: Error finding container 1b27cdb8dd3d6e2506b7c877aa05a88aa04cfd17791fe4b841c72a3d908cadd8: Status 404 returned error can't find the container with id 1b27cdb8dd3d6e2506b7c877aa05a88aa04cfd17791fe4b841c72a3d908cadd8 Aug 04 19:24:07.468246 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.468197 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:07.473628 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.473605 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x"] Aug 04 19:24:07.489674 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.489646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b01960-83e0-4f88-8923-f300677c5aaf-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-xvj9x\" (UID: \"f8b01960-83e0-4f88-8923-f300677c5aaf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" Aug 04 19:24:07.489776 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.489707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5422f\" (UniqueName: \"kubernetes.io/projected/f8b01960-83e0-4f88-8923-f300677c5aaf-kube-api-access-5422f\") pod \"payload-pre-processing-687dbb6cf6-xvj9x\" (UID: \"f8b01960-83e0-4f88-8923-f300677c5aaf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" Aug 04 19:24:07.505783 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.505760 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8lqqg"] Aug 04 19:24:07.507898 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.507884 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" Aug 04 19:24:07.511074 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.511059 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vfh44\"" Aug 04 19:24:07.524400 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.524379 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8lqqg"] Aug 04 19:24:07.590300 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.590277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b01960-83e0-4f88-8923-f300677c5aaf-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-xvj9x\" (UID: \"f8b01960-83e0-4f88-8923-f300677c5aaf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" Aug 04 19:24:07.590455 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.590311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdb4s\" (UniqueName: \"kubernetes.io/projected/5988f76f-b53e-4784-8e0e-3e1599df97e6-kube-api-access-fdb4s\") pod \"payload-processing-5cc6cd444b-8lqqg\" (UID: \"5988f76f-b53e-4784-8e0e-3e1599df97e6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" Aug 04 19:24:07.590455 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.590334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5422f\" (UniqueName: \"kubernetes.io/projected/f8b01960-83e0-4f88-8923-f300677c5aaf-kube-api-access-5422f\") pod \"payload-pre-processing-687dbb6cf6-xvj9x\" (UID: \"f8b01960-83e0-4f88-8923-f300677c5aaf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" Aug 04 19:24:07.590455 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.590350 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5988f76f-b53e-4784-8e0e-3e1599df97e6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8lqqg\" (UID: \"5988f76f-b53e-4784-8e0e-3e1599df97e6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" Aug 04 19:24:07.591039 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.591017 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b01960-83e0-4f88-8923-f300677c5aaf-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-xvj9x\" (UID: \"f8b01960-83e0-4f88-8923-f300677c5aaf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" Aug 04 19:24:07.599719 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.599700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5422f\" (UniqueName: \"kubernetes.io/projected/f8b01960-83e0-4f88-8923-f300677c5aaf-kube-api-access-5422f\") pod \"payload-pre-processing-687dbb6cf6-xvj9x\" (UID: \"f8b01960-83e0-4f88-8923-f300677c5aaf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" Aug 04 19:24:07.691719 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.691691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdb4s\" (UniqueName: \"kubernetes.io/projected/5988f76f-b53e-4784-8e0e-3e1599df97e6-kube-api-access-fdb4s\") pod \"payload-processing-5cc6cd444b-8lqqg\" (UID: \"5988f76f-b53e-4784-8e0e-3e1599df97e6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" Aug 04 19:24:07.691875 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.691727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5988f76f-b53e-4784-8e0e-3e1599df97e6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8lqqg\" (UID: \"5988f76f-b53e-4784-8e0e-3e1599df97e6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" Aug 04 19:24:07.692358 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.692340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5988f76f-b53e-4784-8e0e-3e1599df97e6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8lqqg\" (UID: \"5988f76f-b53e-4784-8e0e-3e1599df97e6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" Aug 04 19:24:07.702916 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.702887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdb4s\" (UniqueName: \"kubernetes.io/projected/5988f76f-b53e-4784-8e0e-3e1599df97e6-kube-api-access-fdb4s\") pod \"payload-processing-5cc6cd444b-8lqqg\" (UID: \"5988f76f-b53e-4784-8e0e-3e1599df97e6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" Aug 04 19:24:07.752430 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.752362 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" event={"ID":"2c659b52-c5e9-483a-8311-d2ae90c856f7","Type":"ContainerStarted","Data":"1b27cdb8dd3d6e2506b7c877aa05a88aa04cfd17791fe4b841c72a3d908cadd8"} Aug 04 19:24:07.784813 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.784788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" Aug 04 19:24:07.817200 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.817173 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" Aug 04 19:24:07.935919 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.935891 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x"] Aug 04 19:24:07.938030 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:24:07.938000 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b01960_83e0_4f88_8923_f300677c5aaf.slice/crio-130df0d34486230f4ef753aa9294cf91c8d708488698a74973a57133f2a7f989 WatchSource:0}: Error finding container 130df0d34486230f4ef753aa9294cf91c8d708488698a74973a57133f2a7f989: Status 404 returned error can't find the container with id 130df0d34486230f4ef753aa9294cf91c8d708488698a74973a57133f2a7f989 Aug 04 19:24:07.959964 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:07.959941 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8lqqg"] Aug 04 19:24:07.962762 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:24:07.962739 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5988f76f_b53e_4784_8e0e_3e1599df97e6.slice/crio-168f598b3d6eeb3e19d70d855c991b2842bb118b9f9ec474b580e83bfea191c3 WatchSource:0}: Error finding container 168f598b3d6eeb3e19d70d855c991b2842bb118b9f9ec474b580e83bfea191c3: Status 404 returned error can't find the container with id 168f598b3d6eeb3e19d70d855c991b2842bb118b9f9ec474b580e83bfea191c3 Aug 04 19:24:08.762029 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:08.761939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" event={"ID":"5988f76f-b53e-4784-8e0e-3e1599df97e6","Type":"ContainerStarted","Data":"168f598b3d6eeb3e19d70d855c991b2842bb118b9f9ec474b580e83bfea191c3"} Aug 04 19:24:08.764355 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:08.764322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" event={"ID":"f8b01960-83e0-4f88-8923-f300677c5aaf","Type":"ContainerStarted","Data":"130df0d34486230f4ef753aa9294cf91c8d708488698a74973a57133f2a7f989"} Aug 04 19:24:10.775264 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:10.775232 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" event={"ID":"5988f76f-b53e-4784-8e0e-3e1599df97e6","Type":"ContainerStarted","Data":"1f9f70f14acb8b345b2526e6c358bfd723ef46f2e701182ea172f050afe7ac08"} Aug 04 19:24:10.775764 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:10.775361 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" Aug 04 19:24:10.776915 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:10.776865 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" event={"ID":"f8b01960-83e0-4f88-8923-f300677c5aaf","Type":"ContainerStarted","Data":"cdb415e03de1bd7dfe3db48bb2aa13bfc0840fda26cf695dae323cb55fc8c3f8"} Aug 04 19:24:10.777056 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:10.776971 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" Aug 04 19:24:10.778334 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:10.778315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" event={"ID":"2c659b52-c5e9-483a-8311-d2ae90c856f7","Type":"ContainerStarted","Data":"cecdd2f7ec25d6cfd2b8088e9ca68f83ce143c855170992f1a1e308a28ed0694"} Aug 04 19:24:10.778517 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:10.778425 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:10.792698 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:10.792655 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" podStartSLOduration=1.85522795 podStartE2EDuration="3.792642591s" totalImagesPullingTime="1.937414641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:24:10.79056327 +0000 UTC m=+495.696163813" watchObservedRunningTime="2026-08-04 19:24:10.792642591 +0000 UTC m=+495.698243137" Aug 04 19:24:10.807233 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:10.807195 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" podStartSLOduration=1.8537453259999999 podStartE2EDuration="3.807182753s" totalImagesPullingTime="1.953437427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:24:10.805411576 +0000 UTC m=+495.711012116" watchObservedRunningTime="2026-08-04 19:24:10.807182753 +0000 UTC m=+495.712783292" Aug 04 19:24:10.819629 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:10.819594 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" podStartSLOduration=2.396176414 podStartE2EDuration="4.819581691s" totalImagesPullingTime="2.423405277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:24:10.818708783 +0000 UTC m=+495.724309326" watchObservedRunningTime="2026-08-04 19:24:10.819581691 +0000 UTC m=+495.725182229" Aug 04 19:24:12.498724 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:12.498677 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:16.789015 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:16.788983 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67c8968587-5lb29" Aug 04 19:24:17.796834 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:17.796804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:17.894378 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:17.894343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:21.785669 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:21.785632 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-xvj9x" Aug 04 19:24:21.786242 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:21.785799 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8lqqg" Aug 04 19:24:21.976795 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:21.976741 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:25.209944 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.209911 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r"] Aug 04 19:24:25.213439 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.213417 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.217491 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.217455 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 04 19:24:25.217642 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.217530 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ppqlf\"" Aug 04 19:24:25.217719 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.217545 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 19:24:25.217719 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.217571 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 19:24:25.225983 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.225960 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r"] Aug 04 19:24:25.253490 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.253442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:25.347217 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.347184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.347420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.347230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj6nd\" (UniqueName: \"kubernetes.io/projected/c1557d34-d920-4ee4-8c25-e4fc135bff4f-kube-api-access-dj6nd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.347420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.347255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1557d34-d920-4ee4-8c25-e4fc135bff4f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.347420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.347319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.347420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.347390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.347420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.347414 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.448568 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.448535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.448568 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.448573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj6nd\" (UniqueName: \"kubernetes.io/projected/c1557d34-d920-4ee4-8c25-e4fc135bff4f-kube-api-access-dj6nd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.448793 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.448592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1557d34-d920-4ee4-8c25-e4fc135bff4f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.448793 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.448642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.448793 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.448706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.448793 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.448742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.449108 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.449088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.449184 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.449123 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.449184 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.449168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.450914 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.450894 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1557d34-d920-4ee4-8c25-e4fc135bff4f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.451189 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.451172 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1557d34-d920-4ee4-8c25-e4fc135bff4f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.458182 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.458161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj6nd\" (UniqueName: \"kubernetes.io/projected/c1557d34-d920-4ee4-8c25-e4fc135bff4f-kube-api-access-dj6nd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r\" (UID: \"c1557d34-d920-4ee4-8c25-e4fc135bff4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.524939 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.524850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:25.664586 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.664342 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r"] Aug 04 19:24:25.666730 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:24:25.666704 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1557d34_d920_4ee4_8c25_e4fc135bff4f.slice/crio-be5a298021c5d0cf01649be4465c75356c36506f025b8cbef3437408b150a0a2 WatchSource:0}: Error finding container be5a298021c5d0cf01649be4465c75356c36506f025b8cbef3437408b150a0a2: Status 404 returned error can't find the container with id be5a298021c5d0cf01649be4465c75356c36506f025b8cbef3437408b150a0a2 Aug 04 19:24:25.847901 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:25.847820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" event={"ID":"c1557d34-d920-4ee4-8c25-e4fc135bff4f","Type":"ContainerStarted","Data":"be5a298021c5d0cf01649be4465c75356c36506f025b8cbef3437408b150a0a2"} Aug 04 19:24:26.853963 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:26.853933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" event={"ID":"c1557d34-d920-4ee4-8c25-e4fc135bff4f","Type":"ContainerStarted","Data":"db65b8868a602328499cf77915a9c2fd1f965d79c1d0fa0a9b8782a803ff0b3a"} Aug 04 19:24:26.875118 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:26.875068 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" podStartSLOduration=0.844899917 podStartE2EDuration="1.87505208s" totalImagesPullingTime="1.030152163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:24:26.873821487 +0000 UTC m=+511.779422038" watchObservedRunningTime="2026-08-04 19:24:26.87505208 +0000 UTC m=+511.780652619" Aug 04 19:24:32.258873 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:32.258836 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:32.347039 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:32.346997 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:35.525308 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:35.525274 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:35.525733 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:35.525324 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:35.537448 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:35.537422 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:35.905809 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:35.905739 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r" Aug 04 19:24:59.703817 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.703738 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k"] Aug 04 19:24:59.707232 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.707216 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.709654 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.709637 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Aug 04 19:24:59.715787 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.715761 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k"] Aug 04 19:24:59.734940 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.734910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.734940 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.734945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.735113 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.734973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93540c46-e980-4c34-8704-8cacc2656efc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.735113 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.735037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.735113 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.735067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtpt\" (UniqueName: \"kubernetes.io/projected/93540c46-e980-4c34-8704-8cacc2656efc-kube-api-access-9mtpt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.735113 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.735092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.806776 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.806745 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:24:59.835921 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.835882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.835921 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.835927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.836090 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.835963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93540c46-e980-4c34-8704-8cacc2656efc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.836090 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.835994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.836090 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.836025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtpt\" (UniqueName: \"kubernetes.io/projected/93540c46-e980-4c34-8704-8cacc2656efc-kube-api-access-9mtpt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.836090 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.836054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.836316 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.836295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.836365 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.836336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.836456 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.836385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.838340 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.838319 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93540c46-e980-4c34-8704-8cacc2656efc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.838604 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.838586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93540c46-e980-4c34-8704-8cacc2656efc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:24:59.844790 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:24:59.844771 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtpt\" (UniqueName: \"kubernetes.io/projected/93540c46-e980-4c34-8704-8cacc2656efc-kube-api-access-9mtpt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lbt2k\" (UID: \"93540c46-e980-4c34-8704-8cacc2656efc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:25:00.018400 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:00.018320 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:25:00.155939 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:00.155916 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k"] Aug 04 19:25:00.157591 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:25:00.157561 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93540c46_e980_4c34_8704_8cacc2656efc.slice/crio-c99e8afb162d74c4d892c75c656d40ed19eab72c54a3a3cce41cfcc3e35e64a5 WatchSource:0}: Error finding container c99e8afb162d74c4d892c75c656d40ed19eab72c54a3a3cce41cfcc3e35e64a5: Status 404 returned error can't find the container with id c99e8afb162d74c4d892c75c656d40ed19eab72c54a3a3cce41cfcc3e35e64a5 Aug 04 19:25:01.011051 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:01.011016 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" event={"ID":"93540c46-e980-4c34-8704-8cacc2656efc","Type":"ContainerStarted","Data":"333ae92b321781ceb56816435f98aec3befd6792ce76098b4bb5a87ef35453ef"} Aug 04 19:25:01.011436 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:01.011063 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" event={"ID":"93540c46-e980-4c34-8704-8cacc2656efc","Type":"ContainerStarted","Data":"c99e8afb162d74c4d892c75c656d40ed19eab72c54a3a3cce41cfcc3e35e64a5"} Aug 04 19:25:01.030902 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:01.030840 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" podStartSLOduration=1.830750793 podStartE2EDuration="2.030821319s" totalImagesPullingTime="200.070526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:25:01.030769099 +0000 UTC m=+545.936369638" watchObservedRunningTime="2026-08-04 19:25:01.030821319 +0000 UTC m=+545.936421859" Aug 04 19:25:03.257150 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:03.257116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:25:10.019274 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:10.019238 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:25:10.019660 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:10.019290 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:25:10.031753 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:10.031731 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:25:10.061804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:10.061778 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lbt2k" Aug 04 19:25:17.041215 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:17.041179 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:25:17.133448 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:17.133416 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:25:24.194590 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:24.194559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:25:24.729759 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:24.729729 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:25:32.198612 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:32.198581 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:25:35.063086 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:35.063055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:25:46.982170 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:46.982140 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl"] Aug 04 19:25:46.985931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:46.985910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:46.988609 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:46.988591 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 04 19:25:46.997514 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:46.997453 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl"] Aug 04 19:25:47.021183 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.021156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:25:47.034236 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.034201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfq67\" (UniqueName: \"kubernetes.io/projected/c7041599-39eb-4372-be28-b4b6032754ab-kube-api-access-mfq67\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.034346 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.034246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.034346 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.034268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.034420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.034350 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.034420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.034374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7041599-39eb-4372-be28-b4b6032754ab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.034420 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.034395 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.135519 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.135482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.135685 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.135587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.135685 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.135618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7041599-39eb-4372-be28-b4b6032754ab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.135685 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.135641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.135822 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.135686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfq67\" (UniqueName: \"kubernetes.io/projected/c7041599-39eb-4372-be28-b4b6032754ab-kube-api-access-mfq67\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.135822 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.135721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.135929 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.135892 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.136072 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.136050 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.136133 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.136085 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.137965 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.137939 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7041599-39eb-4372-be28-b4b6032754ab-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.138243 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.138228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7041599-39eb-4372-be28-b4b6032754ab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.147104 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.147081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfq67\" (UniqueName: \"kubernetes.io/projected/c7041599-39eb-4372-be28-b4b6032754ab-kube-api-access-mfq67\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl\" (UID: \"c7041599-39eb-4372-be28-b4b6032754ab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.295671 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.295582 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:47.432167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:47.432146 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl"] Aug 04 19:25:47.433873 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:25:47.433837 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7041599_39eb_4372_be28_b4b6032754ab.slice/crio-b86d46cf0af2c70664f5b9b1e09fbcd17bd685b22c3caefb369f446371cb01dd WatchSource:0}: Error finding container b86d46cf0af2c70664f5b9b1e09fbcd17bd685b22c3caefb369f446371cb01dd: Status 404 returned error can't find the container with id b86d46cf0af2c70664f5b9b1e09fbcd17bd685b22c3caefb369f446371cb01dd Aug 04 19:25:48.218966 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:48.218935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" event={"ID":"c7041599-39eb-4372-be28-b4b6032754ab","Type":"ContainerStarted","Data":"ffbfba9ab9a8d87cfb2b6e420b4dd711f3e16e707913e756bbf943f1b117fe5b"} Aug 04 19:25:48.218966 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:48.218972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" event={"ID":"c7041599-39eb-4372-be28-b4b6032754ab","Type":"ContainerStarted","Data":"b86d46cf0af2c70664f5b9b1e09fbcd17bd685b22c3caefb369f446371cb01dd"} Aug 04 19:25:48.237507 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:48.237448 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" podStartSLOduration=2.069970664 podStartE2EDuration="2.237432133s" totalImagesPullingTime="167.461469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:25:48.235626571 +0000 UTC m=+593.141227109" watchObservedRunningTime="2026-08-04 19:25:48.237432133 +0000 UTC m=+593.143032672" Aug 04 19:25:55.481159 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:55.481133 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:25:55.482072 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:55.482054 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:25:57.296164 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:57.296136 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:57.296164 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:57.296173 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:57.308792 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:57.308765 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:25:58.277132 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:25:58.277100 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl" Aug 04 19:26:40.517277 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.517242 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74d77cd6ff-hpnmg"] Aug 04 19:26:40.520925 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.520904 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" Aug 04 19:26:40.523762 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.523744 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dxk86\"" Aug 04 19:26:40.524750 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.524733 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 04 19:26:40.529834 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.529814 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74d77cd6ff-hpnmg"] Aug 04 19:26:40.582145 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.582122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10b57ba5-c8bd-4fa8-b242-a3460419e41d-tls-cert\") pod \"authorino-74d77cd6ff-hpnmg\" (UID: \"10b57ba5-c8bd-4fa8-b242-a3460419e41d\") " pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" Aug 04 19:26:40.582262 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.582150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjn4\" (UniqueName: \"kubernetes.io/projected/10b57ba5-c8bd-4fa8-b242-a3460419e41d-kube-api-access-8mjn4\") pod \"authorino-74d77cd6ff-hpnmg\" (UID: \"10b57ba5-c8bd-4fa8-b242-a3460419e41d\") " pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" Aug 04 19:26:40.683658 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.683630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10b57ba5-c8bd-4fa8-b242-a3460419e41d-tls-cert\") pod \"authorino-74d77cd6ff-hpnmg\" (UID: \"10b57ba5-c8bd-4fa8-b242-a3460419e41d\") " pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" Aug 04 19:26:40.683770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.683670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjn4\" (UniqueName: \"kubernetes.io/projected/10b57ba5-c8bd-4fa8-b242-a3460419e41d-kube-api-access-8mjn4\") pod \"authorino-74d77cd6ff-hpnmg\" (UID: \"10b57ba5-c8bd-4fa8-b242-a3460419e41d\") " pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" Aug 04 19:26:40.686126 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.686104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10b57ba5-c8bd-4fa8-b242-a3460419e41d-tls-cert\") pod \"authorino-74d77cd6ff-hpnmg\" (UID: \"10b57ba5-c8bd-4fa8-b242-a3460419e41d\") " pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" Aug 04 19:26:40.691939 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.691919 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjn4\" (UniqueName: \"kubernetes.io/projected/10b57ba5-c8bd-4fa8-b242-a3460419e41d-kube-api-access-8mjn4\") pod \"authorino-74d77cd6ff-hpnmg\" (UID: \"10b57ba5-c8bd-4fa8-b242-a3460419e41d\") " pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" Aug 04 19:26:40.830143 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.830077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" Aug 04 19:26:40.962631 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:40.962605 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74d77cd6ff-hpnmg"] Aug 04 19:26:40.964668 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:26:40.964635 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10b57ba5_c8bd_4fa8_b242_a3460419e41d.slice/crio-7688aca601264efae2334867b22a9943381207eae4340ad614e14b50987a5ce0 WatchSource:0}: Error finding container 7688aca601264efae2334867b22a9943381207eae4340ad614e14b50987a5ce0: Status 404 returned error can't find the container with id 7688aca601264efae2334867b22a9943381207eae4340ad614e14b50987a5ce0 Aug 04 19:26:41.468293 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:41.468264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" event={"ID":"10b57ba5-c8bd-4fa8-b242-a3460419e41d","Type":"ContainerStarted","Data":"7688aca601264efae2334867b22a9943381207eae4340ad614e14b50987a5ce0"} Aug 04 19:26:42.475647 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:42.475603 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" event={"ID":"10b57ba5-c8bd-4fa8-b242-a3460419e41d","Type":"ContainerStarted","Data":"c83405cbc711af45cb195518d0a0863ba90d2ccdb8b3ce8d719d585c587e3ffc"} Aug 04 19:26:42.512838 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:42.512771 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74d77cd6ff-hpnmg" podStartSLOduration=1.4463694679999999 podStartE2EDuration="2.512747406s" totalImagesPullingTime="1.066377938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:26:42.492509814 +0000 UTC m=+647.398110361" watchObservedRunningTime="2026-08-04 19:26:42.512747406 +0000 UTC m=+647.418347946" Aug 04 19:26:43.032961 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:43.032930 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:26:43.129655 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:43.129621 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:26:44.128062 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:44.128022 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:26:44.216344 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:26:44.216311 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:27:31.461553 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.461516 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbff758-5pfkj"] Aug 04 19:27:31.462024 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.461730 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7bbff758-5pfkj" podUID="26b7a1c3-1ac7-4738-b106-e292d29b1213" containerName="manager" containerID="cri-o://a08033bb5b9f26226602c821100e3118e57902cf90ef0818ae1d0b80425c0433" gracePeriod=30 Aug 04 19:27:31.699119 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.699088 2580 generic.go:356] "Generic (PLEG): container finished" podID="26b7a1c3-1ac7-4738-b106-e292d29b1213" containerID="a08033bb5b9f26226602c821100e3118e57902cf90ef0818ae1d0b80425c0433" exitCode=0 Aug 04 19:27:31.699231 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.699160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbff758-5pfkj" event={"ID":"26b7a1c3-1ac7-4738-b106-e292d29b1213","Type":"ContainerDied","Data":"a08033bb5b9f26226602c821100e3118e57902cf90ef0818ae1d0b80425c0433"} Aug 04 19:27:31.699231 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.699213 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbff758-5pfkj" event={"ID":"26b7a1c3-1ac7-4738-b106-e292d29b1213","Type":"ContainerDied","Data":"df9bb13d5fc0f640b02a4bde2074ee140476dfa4d60234e407a3f1d17eef846f"} Aug 04 19:27:31.699306 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.699232 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df9bb13d5fc0f640b02a4bde2074ee140476dfa4d60234e407a3f1d17eef846f" Aug 04 19:27:31.704854 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.704835 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:27:31.722638 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.722587 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms556\" (UniqueName: \"kubernetes.io/projected/26b7a1c3-1ac7-4738-b106-e292d29b1213-kube-api-access-ms556\") pod \"26b7a1c3-1ac7-4738-b106-e292d29b1213\" (UID: \"26b7a1c3-1ac7-4738-b106-e292d29b1213\") " Aug 04 19:27:31.722733 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.722698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b7a1c3-1ac7-4738-b106-e292d29b1213-webhook-cert\") pod \"26b7a1c3-1ac7-4738-b106-e292d29b1213\" (UID: \"26b7a1c3-1ac7-4738-b106-e292d29b1213\") " Aug 04 19:27:31.724921 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.724897 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b7a1c3-1ac7-4738-b106-e292d29b1213-kube-api-access-ms556" (OuterVolumeSpecName: "kube-api-access-ms556") pod "26b7a1c3-1ac7-4738-b106-e292d29b1213" (UID: "26b7a1c3-1ac7-4738-b106-e292d29b1213"). InnerVolumeSpecName "kube-api-access-ms556". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:27:31.725488 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.725430 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b7a1c3-1ac7-4738-b106-e292d29b1213-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "26b7a1c3-1ac7-4738-b106-e292d29b1213" (UID: "26b7a1c3-1ac7-4738-b106-e292d29b1213"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:27:31.823398 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.823371 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26b7a1c3-1ac7-4738-b106-e292d29b1213-webhook-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:27:31.823398 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:31.823394 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms556\" (UniqueName: \"kubernetes.io/projected/26b7a1c3-1ac7-4738-b106-e292d29b1213-kube-api-access-ms556\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:27:32.703551 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:32.703521 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbff758-5pfkj" Aug 04 19:27:32.725867 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:32.725838 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbff758-5pfkj"] Aug 04 19:27:32.733955 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:32.733925 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bbff758-5pfkj"] Aug 04 19:27:33.603425 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:33.603388 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b7a1c3-1ac7-4738-b106-e292d29b1213" path="/var/lib/kubelet/pods/26b7a1c3-1ac7-4738-b106-e292d29b1213/volumes" Aug 04 19:27:40.918566 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:27:40.918519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:28:33.554150 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.554117 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bbff758-2plh2"] Aug 04 19:28:33.554634 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.554511 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b7a1c3-1ac7-4738-b106-e292d29b1213" containerName="manager" Aug 04 19:28:33.554634 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.554524 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b7a1c3-1ac7-4738-b106-e292d29b1213" containerName="manager" Aug 04 19:28:33.554634 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.554590 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b7a1c3-1ac7-4738-b106-e292d29b1213" containerName="manager" Aug 04 19:28:33.557601 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.557585 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:33.561236 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.561215 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vp2lx\"" Aug 04 19:28:33.561360 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.561234 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 19:28:33.561360 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.561262 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 19:28:33.573198 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.573160 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbff758-2plh2"] Aug 04 19:28:33.660323 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.660283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ac157d8-a5e8-4165-a0b7-c04ac799635f-webhook-cert\") pod \"maas-controller-7bbff758-2plh2\" (UID: \"6ac157d8-a5e8-4165-a0b7-c04ac799635f\") " pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:33.660323 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.660335 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd52v\" (UniqueName: \"kubernetes.io/projected/6ac157d8-a5e8-4165-a0b7-c04ac799635f-kube-api-access-cd52v\") pod \"maas-controller-7bbff758-2plh2\" (UID: \"6ac157d8-a5e8-4165-a0b7-c04ac799635f\") " pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:33.761641 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.761591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ac157d8-a5e8-4165-a0b7-c04ac799635f-webhook-cert\") pod \"maas-controller-7bbff758-2plh2\" (UID: \"6ac157d8-a5e8-4165-a0b7-c04ac799635f\") " pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:33.761641 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.761649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd52v\" (UniqueName: \"kubernetes.io/projected/6ac157d8-a5e8-4165-a0b7-c04ac799635f-kube-api-access-cd52v\") pod \"maas-controller-7bbff758-2plh2\" (UID: \"6ac157d8-a5e8-4165-a0b7-c04ac799635f\") " pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:33.764176 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.764151 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ac157d8-a5e8-4165-a0b7-c04ac799635f-webhook-cert\") pod \"maas-controller-7bbff758-2plh2\" (UID: \"6ac157d8-a5e8-4165-a0b7-c04ac799635f\") " pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:33.771112 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.771088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd52v\" (UniqueName: \"kubernetes.io/projected/6ac157d8-a5e8-4165-a0b7-c04ac799635f-kube-api-access-cd52v\") pod \"maas-controller-7bbff758-2plh2\" (UID: \"6ac157d8-a5e8-4165-a0b7-c04ac799635f\") " pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:33.867518 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:33.867403 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:34.003209 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:34.003187 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbff758-2plh2"] Aug 04 19:28:34.005719 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:28:34.005691 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac157d8_a5e8_4165_a0b7_c04ac799635f.slice/crio-65a4cacfc2adb80a023edf3670fded40012be2c5b6f418946ecc9455783d40c1 WatchSource:0}: Error finding container 65a4cacfc2adb80a023edf3670fded40012be2c5b6f418946ecc9455783d40c1: Status 404 returned error can't find the container with id 65a4cacfc2adb80a023edf3670fded40012be2c5b6f418946ecc9455783d40c1 Aug 04 19:28:34.007169 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:34.007154 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:28:34.990602 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:34.990569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbff758-2plh2" event={"ID":"6ac157d8-a5e8-4165-a0b7-c04ac799635f","Type":"ContainerStarted","Data":"d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055"} Aug 04 19:28:34.990978 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:34.990621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbff758-2plh2" event={"ID":"6ac157d8-a5e8-4165-a0b7-c04ac799635f","Type":"ContainerStarted","Data":"65a4cacfc2adb80a023edf3670fded40012be2c5b6f418946ecc9455783d40c1"} Aug 04 19:28:34.990978 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:34.990650 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:35.009917 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:35.009858 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bbff758-2plh2" podStartSLOduration=1.546973309 podStartE2EDuration="2.009840993s" totalImagesPullingTime="462.867684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:28:35.009396843 +0000 UTC m=+759.914997383" watchObservedRunningTime="2026-08-04 19:28:35.009840993 +0000 UTC m=+759.915441532" Aug 04 19:28:45.997707 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:45.997679 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:50.532702 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:50.532668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:28:50.631447 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:50.631411 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbff758-2plh2"] Aug 04 19:28:50.632003 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:50.631951 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7bbff758-2plh2" podUID="6ac157d8-a5e8-4165-a0b7-c04ac799635f" containerName="manager" containerID="cri-o://d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055" gracePeriod=30 Aug 04 19:28:50.880117 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:50.880094 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:51.007666 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.007633 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ac157d8-a5e8-4165-a0b7-c04ac799635f-webhook-cert\") pod \"6ac157d8-a5e8-4165-a0b7-c04ac799635f\" (UID: \"6ac157d8-a5e8-4165-a0b7-c04ac799635f\") " Aug 04 19:28:51.007825 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.007682 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd52v\" (UniqueName: \"kubernetes.io/projected/6ac157d8-a5e8-4165-a0b7-c04ac799635f-kube-api-access-cd52v\") pod \"6ac157d8-a5e8-4165-a0b7-c04ac799635f\" (UID: \"6ac157d8-a5e8-4165-a0b7-c04ac799635f\") " Aug 04 19:28:51.009778 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.009747 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac157d8-a5e8-4165-a0b7-c04ac799635f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6ac157d8-a5e8-4165-a0b7-c04ac799635f" (UID: "6ac157d8-a5e8-4165-a0b7-c04ac799635f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:28:51.009886 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.009831 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac157d8-a5e8-4165-a0b7-c04ac799635f-kube-api-access-cd52v" (OuterVolumeSpecName: "kube-api-access-cd52v") pod "6ac157d8-a5e8-4165-a0b7-c04ac799635f" (UID: "6ac157d8-a5e8-4165-a0b7-c04ac799635f"). InnerVolumeSpecName "kube-api-access-cd52v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:28:51.065824 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.065753 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ac157d8-a5e8-4165-a0b7-c04ac799635f" containerID="d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055" exitCode=0 Aug 04 19:28:51.065824 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.065802 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbff758-2plh2" Aug 04 19:28:51.066001 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.065839 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbff758-2plh2" event={"ID":"6ac157d8-a5e8-4165-a0b7-c04ac799635f","Type":"ContainerDied","Data":"d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055"} Aug 04 19:28:51.066001 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.065900 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbff758-2plh2" event={"ID":"6ac157d8-a5e8-4165-a0b7-c04ac799635f","Type":"ContainerDied","Data":"65a4cacfc2adb80a023edf3670fded40012be2c5b6f418946ecc9455783d40c1"} Aug 04 19:28:51.066001 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.065924 2580 scope.go:117] "RemoveContainer" containerID="d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055" Aug 04 19:28:51.074523 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.074504 2580 scope.go:117] "RemoveContainer" containerID="d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055" Aug 04 19:28:51.074786 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:28:51.074769 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055\": container with ID starting with d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055 not found: ID does not exist" containerID="d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055" Aug 04 19:28:51.074833 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.074793 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055"} err="failed to get container status \"d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055\": rpc error: code = NotFound desc = could not find container \"d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055\": container with ID starting with d4588c86b0aa186af60c2562de430a81f9f30e702a6e796de5e4849adec17055 not found: ID does not exist" Aug 04 19:28:51.086125 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.086103 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbff758-2plh2"] Aug 04 19:28:51.091520 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.091500 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bbff758-2plh2"] Aug 04 19:28:51.109193 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.109170 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ac157d8-a5e8-4165-a0b7-c04ac799635f-webhook-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:28:51.109193 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.109190 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd52v\" (UniqueName: \"kubernetes.io/projected/6ac157d8-a5e8-4165-a0b7-c04ac799635f-kube-api-access-cd52v\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:28:51.604150 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.604118 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac157d8-a5e8-4165-a0b7-c04ac799635f" path="/var/lib/kubelet/pods/6ac157d8-a5e8-4165-a0b7-c04ac799635f/volumes" Aug 04 19:28:51.782268 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.782232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:28:51.894861 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.894785 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bbff758-f9jws"] Aug 04 19:28:51.895169 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.895157 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac157d8-a5e8-4165-a0b7-c04ac799635f" containerName="manager" Aug 04 19:28:51.895209 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.895170 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac157d8-a5e8-4165-a0b7-c04ac799635f" containerName="manager" Aug 04 19:28:51.895241 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.895235 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac157d8-a5e8-4165-a0b7-c04ac799635f" containerName="manager" Aug 04 19:28:51.899574 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.899560 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbff758-f9jws" Aug 04 19:28:51.903249 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.903225 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 19:28:51.903368 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.903225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vp2lx\"" Aug 04 19:28:51.903368 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.903229 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 19:28:51.910168 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:51.910148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbff758-f9jws"] Aug 04 19:28:52.017310 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:52.017279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjxzn\" (UniqueName: \"kubernetes.io/projected/2b9a94fb-e69b-4075-affe-0c5f2690c042-kube-api-access-fjxzn\") pod \"maas-controller-7bbff758-f9jws\" (UID: \"2b9a94fb-e69b-4075-affe-0c5f2690c042\") " pod="opendatahub/maas-controller-7bbff758-f9jws" Aug 04 19:28:52.017310 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:52.017314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b9a94fb-e69b-4075-affe-0c5f2690c042-webhook-cert\") pod \"maas-controller-7bbff758-f9jws\" (UID: \"2b9a94fb-e69b-4075-affe-0c5f2690c042\") " pod="opendatahub/maas-controller-7bbff758-f9jws" Aug 04 19:28:52.118220 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:52.118188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxzn\" (UniqueName: \"kubernetes.io/projected/2b9a94fb-e69b-4075-affe-0c5f2690c042-kube-api-access-fjxzn\") pod \"maas-controller-7bbff758-f9jws\" (UID: \"2b9a94fb-e69b-4075-affe-0c5f2690c042\") " pod="opendatahub/maas-controller-7bbff758-f9jws" Aug 04 19:28:52.118220 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:52.118221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b9a94fb-e69b-4075-affe-0c5f2690c042-webhook-cert\") pod \"maas-controller-7bbff758-f9jws\" (UID: \"2b9a94fb-e69b-4075-affe-0c5f2690c042\") " pod="opendatahub/maas-controller-7bbff758-f9jws" Aug 04 19:28:52.120915 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:52.120884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b9a94fb-e69b-4075-affe-0c5f2690c042-webhook-cert\") pod \"maas-controller-7bbff758-f9jws\" (UID: \"2b9a94fb-e69b-4075-affe-0c5f2690c042\") " pod="opendatahub/maas-controller-7bbff758-f9jws" Aug 04 19:28:52.126879 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:52.126856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxzn\" (UniqueName: \"kubernetes.io/projected/2b9a94fb-e69b-4075-affe-0c5f2690c042-kube-api-access-fjxzn\") pod \"maas-controller-7bbff758-f9jws\" (UID: \"2b9a94fb-e69b-4075-affe-0c5f2690c042\") " pod="opendatahub/maas-controller-7bbff758-f9jws" Aug 04 19:28:52.210830 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:52.210802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbff758-f9jws" Aug 04 19:28:52.346326 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:52.346295 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbff758-f9jws"] Aug 04 19:28:52.348961 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:28:52.348927 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9a94fb_e69b_4075_affe_0c5f2690c042.slice/crio-bd06286c28d6ebcafe656869b752b40994301302ee6f23c6aa3cc879b23e51dc WatchSource:0}: Error finding container bd06286c28d6ebcafe656869b752b40994301302ee6f23c6aa3cc879b23e51dc: Status 404 returned error can't find the container with id bd06286c28d6ebcafe656869b752b40994301302ee6f23c6aa3cc879b23e51dc Aug 04 19:28:53.077961 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:53.077928 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbff758-f9jws" event={"ID":"2b9a94fb-e69b-4075-affe-0c5f2690c042","Type":"ContainerStarted","Data":"110692510cabe6b2a6e6f8313256e5b8cdf4171a87b8a0688822286d0eb5270b"} Aug 04 19:28:53.078399 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:53.078075 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bbff758-f9jws" Aug 04 19:28:53.078399 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:53.078125 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbff758-f9jws" event={"ID":"2b9a94fb-e69b-4075-affe-0c5f2690c042","Type":"ContainerStarted","Data":"bd06286c28d6ebcafe656869b752b40994301302ee6f23c6aa3cc879b23e51dc"} Aug 04 19:28:53.097835 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:28:53.097794 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bbff758-f9jws" podStartSLOduration=1.779998617 podStartE2EDuration="2.097781702s" totalImagesPullingTime="317.783085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:28:53.096278344 +0000 UTC m=+778.001878894" watchObservedRunningTime="2026-08-04 19:28:53.097781702 +0000 UTC m=+778.003382240" Aug 04 19:29:04.085251 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:29:04.085222 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bbff758-f9jws" Aug 04 19:29:47.309877 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:29:47.309802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:29:47.424107 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:29:47.424064 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:29:55.538592 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:29:55.538560 2580 scope.go:117] "RemoveContainer" containerID="a08033bb5b9f26226602c821100e3118e57902cf90ef0818ae1d0b80425c0433" Aug 04 19:29:57.039748 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:29:57.039691 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:29:57.126785 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:29:57.126751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:29:57.279340 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:29:57.279308 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:30:03.126710 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:30:03.126667 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:30:03.236753 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:30:03.236700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:30:03.783126 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:30:03.783096 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:30:03.849687 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:30:03.849656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:30:09.440486 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:30:09.440442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:30:09.502190 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:30:09.502143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:30:55.508302 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:30:55.508277 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:30:55.510005 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:30:55.509985 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:31:56.653482 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:31:56.653430 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:31:57.727303 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:31:57.727268 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:32:07.519035 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:32:07.518999 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:32:07.620107 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:32:07.620052 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:32:07.777386 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:32:07.777297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:32:08.466581 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:32:08.466541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:32:23.650337 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:32:23.650252 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:35:11.620204 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:35:11.620171 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:35:11.706267 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:35:11.706236 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:35:55.543964 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:35:55.543931 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:35:55.546104 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:35:55.546080 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:37:28.442122 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:37:28.442085 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:39:06.844991 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:06.844961 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:39:24.490071 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:24.490035 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:39:26.622022 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:26.621986 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:39:26.684363 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:26.684315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:39:36.088442 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:36.088410 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:39:43.845016 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:43.844978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:39:46.859017 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:46.858936 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb"] Aug 04 19:39:46.862925 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:46.862905 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:46.865280 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:46.865261 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3fe4ffd9-kserve-self-signed-certs\"" Aug 04 19:39:46.879269 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:46.879245 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb"] Aug 04 19:39:46.902947 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:46.902916 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:39:47.037184 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.037145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rjt\" (UniqueName: \"kubernetes.io/projected/2920d6c3-79ff-4f5b-8735-7176d75420bc-kube-api-access-54rjt\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.037340 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.037195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-dshm\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.037340 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.037219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-tmp-dir\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.037340 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.037299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-model-cache\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.037340 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.037326 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-home\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.037499 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.037346 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2920d6c3-79ff-4f5b-8735-7176d75420bc-tls-certs\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.138177 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.138095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-model-cache\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.138177 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.138137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-home\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.138177 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.138158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2920d6c3-79ff-4f5b-8735-7176d75420bc-tls-certs\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.138436 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.138209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rjt\" (UniqueName: \"kubernetes.io/projected/2920d6c3-79ff-4f5b-8735-7176d75420bc-kube-api-access-54rjt\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.138436 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.138239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-dshm\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.138436 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.138264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-tmp-dir\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.138981 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.138944 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-model-cache\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.139129 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.139064 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-tmp-dir\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.139218 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.139197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-home\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.144605 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.141437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-dshm\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.144605 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.143543 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2920d6c3-79ff-4f5b-8735-7176d75420bc-tls-certs\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.158536 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.158499 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rjt\" (UniqueName: \"kubernetes.io/projected/2920d6c3-79ff-4f5b-8735-7176d75420bc-kube-api-access-54rjt\") pod \"e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.173089 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.173058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:47.324011 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:47.323984 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb"] Aug 04 19:39:47.325211 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:39:47.325184 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2920d6c3_79ff_4f5b_8735_7176d75420bc.slice/crio-5b244dbc72de809f08f3682a5dd9e6b5f4d2eb6e1070d5e5f6519d9565d794fd WatchSource:0}: Error finding container 5b244dbc72de809f08f3682a5dd9e6b5f4d2eb6e1070d5e5f6519d9565d794fd: Status 404 returned error can't find the container with id 5b244dbc72de809f08f3682a5dd9e6b5f4d2eb6e1070d5e5f6519d9565d794fd Aug 04 19:39:48.099089 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:48.099053 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" event={"ID":"2920d6c3-79ff-4f5b-8735-7176d75420bc","Type":"ContainerStarted","Data":"b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed"} Aug 04 19:39:48.099089 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:48.099093 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" event={"ID":"2920d6c3-79ff-4f5b-8735-7176d75420bc","Type":"ContainerStarted","Data":"5b244dbc72de809f08f3682a5dd9e6b5f4d2eb6e1070d5e5f6519d9565d794fd"} Aug 04 19:39:57.173553 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:57.173523 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:57.173979 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:57.173567 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:57.186780 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:57.186757 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:39:57.201259 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:57.201217 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" podStartSLOduration=11.201204894 podStartE2EDuration="11.201204894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:39:48.118878759 +0000 UTC m=+1433.024479299" watchObservedRunningTime="2026-08-04 19:39:57.201204894 +0000 UTC m=+1442.106805434" Aug 04 19:39:58.155911 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:39:58.155886 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:40:06.489939 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:06.489909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:40:07.711719 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:07.711682 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb"] Aug 04 19:40:07.735628 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:07.735596 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:40:22.732839 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:22.732779 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" podUID="2920d6c3-79ff-4f5b-8735-7176d75420bc" containerName="main" containerID="cri-o://b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed" gracePeriod=45 Aug 04 19:40:22.980402 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:22.980374 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:40:23.053101 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053039 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-dshm\") pod \"2920d6c3-79ff-4f5b-8735-7176d75420bc\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " Aug 04 19:40:23.053101 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053066 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-home\") pod \"2920d6c3-79ff-4f5b-8735-7176d75420bc\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " Aug 04 19:40:23.053101 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053081 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-model-cache\") pod \"2920d6c3-79ff-4f5b-8735-7176d75420bc\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " Aug 04 19:40:23.053342 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053156 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2920d6c3-79ff-4f5b-8735-7176d75420bc-tls-certs\") pod \"2920d6c3-79ff-4f5b-8735-7176d75420bc\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " Aug 04 19:40:23.053342 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053208 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-tmp-dir\") pod \"2920d6c3-79ff-4f5b-8735-7176d75420bc\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " Aug 04 19:40:23.053342 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053235 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54rjt\" (UniqueName: \"kubernetes.io/projected/2920d6c3-79ff-4f5b-8735-7176d75420bc-kube-api-access-54rjt\") pod \"2920d6c3-79ff-4f5b-8735-7176d75420bc\" (UID: \"2920d6c3-79ff-4f5b-8735-7176d75420bc\") " Aug 04 19:40:23.053523 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053352 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-home" (OuterVolumeSpecName: "home") pod "2920d6c3-79ff-4f5b-8735-7176d75420bc" (UID: "2920d6c3-79ff-4f5b-8735-7176d75420bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:40:23.053523 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053396 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "2920d6c3-79ff-4f5b-8735-7176d75420bc" (UID: "2920d6c3-79ff-4f5b-8735-7176d75420bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:40:23.053523 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053458 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2920d6c3-79ff-4f5b-8735-7176d75420bc" (UID: "2920d6c3-79ff-4f5b-8735-7176d75420bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:40:23.053677 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053558 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-tmp-dir\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:40:23.053677 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053569 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-home\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:40:23.053677 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.053579 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-model-cache\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:40:23.055374 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.055350 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-dshm" (OuterVolumeSpecName: "dshm") pod "2920d6c3-79ff-4f5b-8735-7176d75420bc" (UID: "2920d6c3-79ff-4f5b-8735-7176d75420bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:40:23.055506 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.055429 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2920d6c3-79ff-4f5b-8735-7176d75420bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2920d6c3-79ff-4f5b-8735-7176d75420bc" (UID: "2920d6c3-79ff-4f5b-8735-7176d75420bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:40:23.055506 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.055446 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2920d6c3-79ff-4f5b-8735-7176d75420bc-kube-api-access-54rjt" (OuterVolumeSpecName: "kube-api-access-54rjt") pod "2920d6c3-79ff-4f5b-8735-7176d75420bc" (UID: "2920d6c3-79ff-4f5b-8735-7176d75420bc"). InnerVolumeSpecName "kube-api-access-54rjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:40:23.155002 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.154972 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2920d6c3-79ff-4f5b-8735-7176d75420bc-dshm\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:40:23.155002 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.154998 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2920d6c3-79ff-4f5b-8735-7176d75420bc-tls-certs\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:40:23.155175 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.155009 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54rjt\" (UniqueName: \"kubernetes.io/projected/2920d6c3-79ff-4f5b-8735-7176d75420bc-kube-api-access-54rjt\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:40:23.264748 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.264715 2580 generic.go:356] "Generic (PLEG): container finished" podID="2920d6c3-79ff-4f5b-8735-7176d75420bc" containerID="b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed" exitCode=0 Aug 04 19:40:23.264913 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.264758 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" Aug 04 19:40:23.264913 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.264774 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" event={"ID":"2920d6c3-79ff-4f5b-8735-7176d75420bc","Type":"ContainerDied","Data":"b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed"} Aug 04 19:40:23.264913 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.264813 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb" event={"ID":"2920d6c3-79ff-4f5b-8735-7176d75420bc","Type":"ContainerDied","Data":"5b244dbc72de809f08f3682a5dd9e6b5f4d2eb6e1070d5e5f6519d9565d794fd"} Aug 04 19:40:23.264913 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.264834 2580 scope.go:117] "RemoveContainer" containerID="b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed" Aug 04 19:40:23.274242 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.274220 2580 scope.go:117] "RemoveContainer" containerID="b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed" Aug 04 19:40:23.274519 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:40:23.274499 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed\": container with ID starting with b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed not found: ID does not exist" containerID="b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed" Aug 04 19:40:23.274616 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.274530 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed"} err="failed to get container status \"b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed\": rpc error: code = NotFound desc = could not find container \"b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed\": container with ID starting with b517779243e7443a6e973dbcc8548c4196fd574459d56192f8ae70883f87e8ed not found: ID does not exist" Aug 04 19:40:23.287751 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.287730 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb"] Aug 04 19:40:23.292073 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.292053 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3fe4ffd9-kserve-6bf7858b67-gh2fb"] Aug 04 19:40:23.608818 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:23.608785 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2920d6c3-79ff-4f5b-8735-7176d75420bc" path="/var/lib/kubelet/pods/2920d6c3-79ff-4f5b-8735-7176d75420bc/volumes" Aug 04 19:40:55.573594 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:55.573565 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:40:55.575835 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:40:55.575816 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:42:56.187576 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:42:56.187491 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:42:57.441813 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:42:57.441772 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:43:25.773931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:43:25.773891 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:43:51.779585 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:43:51.779539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:43:54.785328 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:43:54.785297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:43:54.878780 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:43:54.878745 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:43:55.253270 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:43:55.253225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:00.921382 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:00.921345 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:01.021179 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:01.021145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:01.142485 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:01.142428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:01.429189 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:01.429157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:01.560612 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:01.560575 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:03.296331 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:03.296285 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:03.351923 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:03.351895 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:04.436389 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:04.436346 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:04.550870 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:04.550828 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:04.923834 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:04.923785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:05.022864 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.022824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:05.027248 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.027225 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg"] Aug 04 19:44:05.027628 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.027614 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2920d6c3-79ff-4f5b-8735-7176d75420bc" containerName="main" Aug 04 19:44:05.027673 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.027631 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2920d6c3-79ff-4f5b-8735-7176d75420bc" containerName="main" Aug 04 19:44:05.027708 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.027704 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2920d6c3-79ff-4f5b-8735-7176d75420bc" containerName="main" Aug 04 19:44:05.030656 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.030641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:05.033011 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.032993 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-89a8e771\"" Aug 04 19:44:05.040258 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.040236 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg"] Aug 04 19:44:05.066127 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.066093 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz"] Aug 04 19:44:05.070193 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.070176 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:05.072815 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.072795 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-89a8e771-dockercfg-4j6hz\"" Aug 04 19:44:05.076876 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.076856 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz"] Aug 04 19:44:05.100161 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.100139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301bad9c-1f44-4440-a184-655b0c65e947-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg\" (UID: \"301bad9c-1f44-4440-a184-655b0c65e947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:05.100290 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.100172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwld2\" (UniqueName: \"kubernetes.io/projected/301bad9c-1f44-4440-a184-655b0c65e947-kube-api-access-rwld2\") pod \"payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg\" (UID: \"301bad9c-1f44-4440-a184-655b0c65e947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:05.100290 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.100193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c750094-bcab-4d96-be9e-f7aabe3a3202-plugins-config-volume\") pod \"payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz\" (UID: \"6c750094-bcab-4d96-be9e-f7aabe3a3202\") " pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:05.100290 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.100275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5nf\" (UniqueName: \"kubernetes.io/projected/6c750094-bcab-4d96-be9e-f7aabe3a3202-kube-api-access-2f5nf\") pod \"payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz\" (UID: \"6c750094-bcab-4d96-be9e-f7aabe3a3202\") " pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:05.201598 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.201553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301bad9c-1f44-4440-a184-655b0c65e947-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg\" (UID: \"301bad9c-1f44-4440-a184-655b0c65e947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:05.201598 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.201601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwld2\" (UniqueName: \"kubernetes.io/projected/301bad9c-1f44-4440-a184-655b0c65e947-kube-api-access-rwld2\") pod \"payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg\" (UID: \"301bad9c-1f44-4440-a184-655b0c65e947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:05.201838 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.201629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c750094-bcab-4d96-be9e-f7aabe3a3202-plugins-config-volume\") pod \"payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz\" (UID: \"6c750094-bcab-4d96-be9e-f7aabe3a3202\") " pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:05.201838 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.201663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5nf\" (UniqueName: \"kubernetes.io/projected/6c750094-bcab-4d96-be9e-f7aabe3a3202-kube-api-access-2f5nf\") pod \"payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz\" (UID: \"6c750094-bcab-4d96-be9e-f7aabe3a3202\") " pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:05.202329 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.202309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301bad9c-1f44-4440-a184-655b0c65e947-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg\" (UID: \"301bad9c-1f44-4440-a184-655b0c65e947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:05.202402 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.202360 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c750094-bcab-4d96-be9e-f7aabe3a3202-plugins-config-volume\") pod \"payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz\" (UID: \"6c750094-bcab-4d96-be9e-f7aabe3a3202\") " pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:05.210035 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.210011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwld2\" (UniqueName: \"kubernetes.io/projected/301bad9c-1f44-4440-a184-655b0c65e947-kube-api-access-rwld2\") pod \"payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg\" (UID: \"301bad9c-1f44-4440-a184-655b0c65e947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:05.210182 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.210160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5nf\" (UniqueName: \"kubernetes.io/projected/6c750094-bcab-4d96-be9e-f7aabe3a3202-kube-api-access-2f5nf\") pod \"payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz\" (UID: \"6c750094-bcab-4d96-be9e-f7aabe3a3202\") " pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:05.342099 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.342067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:05.382564 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.382534 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:05.490769 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.490739 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg"] Aug 04 19:44:05.492725 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:44:05.492696 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301bad9c_1f44_4440_a184_655b0c65e947.slice/crio-c36c32deca5d60291980454ea0cfe582491c84fad02cbb144b30befb683121a9 WatchSource:0}: Error finding container c36c32deca5d60291980454ea0cfe582491c84fad02cbb144b30befb683121a9: Status 404 returned error can't find the container with id c36c32deca5d60291980454ea0cfe582491c84fad02cbb144b30befb683121a9 Aug 04 19:44:05.534401 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:05.534292 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz"] Aug 04 19:44:05.555610 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:44:05.555582 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c750094_bcab_4d96_be9e_f7aabe3a3202.slice/crio-99b746f62846bd43db57be9519d60f7c1db199878d756e47337a84416acd0fbf WatchSource:0}: Error finding container 99b746f62846bd43db57be9519d60f7c1db199878d756e47337a84416acd0fbf: Status 404 returned error can't find the container with id 99b746f62846bd43db57be9519d60f7c1db199878d756e47337a84416acd0fbf Aug 04 19:44:06.316102 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:06.316067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" event={"ID":"301bad9c-1f44-4440-a184-655b0c65e947","Type":"ContainerStarted","Data":"3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9"} Aug 04 19:44:06.316317 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:06.316123 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" event={"ID":"301bad9c-1f44-4440-a184-655b0c65e947","Type":"ContainerStarted","Data":"c36c32deca5d60291980454ea0cfe582491c84fad02cbb144b30befb683121a9"} Aug 04 19:44:06.316317 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:06.316178 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:06.317640 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:06.317615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" event={"ID":"6c750094-bcab-4d96-be9e-f7aabe3a3202","Type":"ContainerStarted","Data":"8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0"} Aug 04 19:44:06.317777 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:06.317660 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:06.317777 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:06.317683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" event={"ID":"6c750094-bcab-4d96-be9e-f7aabe3a3202","Type":"ContainerStarted","Data":"99b746f62846bd43db57be9519d60f7c1db199878d756e47337a84416acd0fbf"} Aug 04 19:44:06.335433 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:06.333783 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" podStartSLOduration=1.33376195 podStartE2EDuration="1.33376195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:44:06.330756923 +0000 UTC m=+1691.236357462" watchObservedRunningTime="2026-08-04 19:44:06.33376195 +0000 UTC m=+1691.239362489" Aug 04 19:44:06.352689 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:06.352645 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" podStartSLOduration=1.352630426 podStartE2EDuration="1.352630426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:44:06.350618968 +0000 UTC m=+1691.256219498" watchObservedRunningTime="2026-08-04 19:44:06.352630426 +0000 UTC m=+1691.258230964" Aug 04 19:44:17.324214 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:17.324124 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:17.324626 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:17.324290 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:35.078356 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.078315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:35.112584 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.111943 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz"] Aug 04 19:44:35.112584 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.112178 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" podUID="6c750094-bcab-4d96-be9e-f7aabe3a3202" containerName="payload-processing" containerID="cri-o://8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0" gracePeriod=30 Aug 04 19:44:35.118110 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.118075 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg"] Aug 04 19:44:35.118292 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.118252 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" podUID="301bad9c-1f44-4440-a184-655b0c65e947" containerName="payload-pre-processing" containerID="cri-o://3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9" gracePeriod=30 Aug 04 19:44:35.150673 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.150641 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:35.211034 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.210994 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:35.315885 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.315825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:35.391230 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.391205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:35.407878 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.407852 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:35.411006 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.410983 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:35.463987 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.463954 2580 generic.go:356] "Generic (PLEG): container finished" podID="6c750094-bcab-4d96-be9e-f7aabe3a3202" containerID="8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0" exitCode=0 Aug 04 19:44:35.464152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.463997 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" Aug 04 19:44:35.464152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.464085 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" event={"ID":"6c750094-bcab-4d96-be9e-f7aabe3a3202","Type":"ContainerDied","Data":"8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0"} Aug 04 19:44:35.464152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.464129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz" event={"ID":"6c750094-bcab-4d96-be9e-f7aabe3a3202","Type":"ContainerDied","Data":"99b746f62846bd43db57be9519d60f7c1db199878d756e47337a84416acd0fbf"} Aug 04 19:44:35.464318 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.464157 2580 scope.go:117] "RemoveContainer" containerID="8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0" Aug 04 19:44:35.465570 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.465549 2580 generic.go:356] "Generic (PLEG): container finished" podID="301bad9c-1f44-4440-a184-655b0c65e947" containerID="3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9" exitCode=0 Aug 04 19:44:35.465688 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.465591 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" Aug 04 19:44:35.465688 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.465591 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" event={"ID":"301bad9c-1f44-4440-a184-655b0c65e947","Type":"ContainerDied","Data":"3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9"} Aug 04 19:44:35.465791 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.465733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg" event={"ID":"301bad9c-1f44-4440-a184-655b0c65e947","Type":"ContainerDied","Data":"c36c32deca5d60291980454ea0cfe582491c84fad02cbb144b30befb683121a9"} Aug 04 19:44:35.472955 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.472936 2580 scope.go:117] "RemoveContainer" containerID="8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0" Aug 04 19:44:35.473253 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:44:35.473234 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0\": container with ID starting with 8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0 not found: ID does not exist" containerID="8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0" Aug 04 19:44:35.473365 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.473258 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0"} err="failed to get container status \"8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0\": rpc error: code = NotFound desc = could not find container \"8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0\": container with ID starting with 8d96c061ca1a229f7bf987c2507d0a5080e8da3983e47d4af59c52e386e74ba0 not found: ID does not exist" Aug 04 19:44:35.473365 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.473277 2580 scope.go:117] "RemoveContainer" containerID="3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9" Aug 04 19:44:35.481049 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.481030 2580 scope.go:117] "RemoveContainer" containerID="3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9" Aug 04 19:44:35.481315 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:44:35.481298 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9\": container with ID starting with 3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9 not found: ID does not exist" containerID="3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9" Aug 04 19:44:35.481381 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.481324 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9"} err="failed to get container status \"3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9\": rpc error: code = NotFound desc = could not find container \"3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9\": container with ID starting with 3933a5f25e5c55466e5a02d0252c3c7b7bb8fe7ef49cea82e6028701b7c152b9 not found: ID does not exist" Aug 04 19:44:35.582542 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.582513 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f5nf\" (UniqueName: \"kubernetes.io/projected/6c750094-bcab-4d96-be9e-f7aabe3a3202-kube-api-access-2f5nf\") pod \"6c750094-bcab-4d96-be9e-f7aabe3a3202\" (UID: \"6c750094-bcab-4d96-be9e-f7aabe3a3202\") " Aug 04 19:44:35.582717 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.582575 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwld2\" (UniqueName: \"kubernetes.io/projected/301bad9c-1f44-4440-a184-655b0c65e947-kube-api-access-rwld2\") pod \"301bad9c-1f44-4440-a184-655b0c65e947\" (UID: \"301bad9c-1f44-4440-a184-655b0c65e947\") " Aug 04 19:44:35.582717 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.582630 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301bad9c-1f44-4440-a184-655b0c65e947-plugins-config-volume\") pod \"301bad9c-1f44-4440-a184-655b0c65e947\" (UID: \"301bad9c-1f44-4440-a184-655b0c65e947\") " Aug 04 19:44:35.582717 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.582661 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c750094-bcab-4d96-be9e-f7aabe3a3202-plugins-config-volume\") pod \"6c750094-bcab-4d96-be9e-f7aabe3a3202\" (UID: \"6c750094-bcab-4d96-be9e-f7aabe3a3202\") " Aug 04 19:44:35.583012 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.582987 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301bad9c-1f44-4440-a184-655b0c65e947-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "301bad9c-1f44-4440-a184-655b0c65e947" (UID: "301bad9c-1f44-4440-a184-655b0c65e947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:44:35.583113 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.583044 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c750094-bcab-4d96-be9e-f7aabe3a3202-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c750094-bcab-4d96-be9e-f7aabe3a3202" (UID: "6c750094-bcab-4d96-be9e-f7aabe3a3202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:44:35.584948 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.584929 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c750094-bcab-4d96-be9e-f7aabe3a3202-kube-api-access-2f5nf" (OuterVolumeSpecName: "kube-api-access-2f5nf") pod "6c750094-bcab-4d96-be9e-f7aabe3a3202" (UID: "6c750094-bcab-4d96-be9e-f7aabe3a3202"). InnerVolumeSpecName "kube-api-access-2f5nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:44:35.585017 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.585002 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301bad9c-1f44-4440-a184-655b0c65e947-kube-api-access-rwld2" (OuterVolumeSpecName: "kube-api-access-rwld2") pod "301bad9c-1f44-4440-a184-655b0c65e947" (UID: "301bad9c-1f44-4440-a184-655b0c65e947"). InnerVolumeSpecName "kube-api-access-rwld2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:44:35.631227 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.631202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:35.683967 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.683899 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwld2\" (UniqueName: \"kubernetes.io/projected/301bad9c-1f44-4440-a184-655b0c65e947-kube-api-access-rwld2\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:44:35.683967 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.683925 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/301bad9c-1f44-4440-a184-655b0c65e947-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:44:35.683967 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.683935 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c750094-bcab-4d96-be9e-f7aabe3a3202-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:44:35.683967 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.683944 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f5nf\" (UniqueName: \"kubernetes.io/projected/6c750094-bcab-4d96-be9e-f7aabe3a3202-kube-api-access-2f5nf\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:44:35.780889 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.780855 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg"] Aug 04 19:44:35.784823 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.784800 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89a8e771-b55cbf64-jfbgg"] Aug 04 19:44:35.797889 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.797857 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz"] Aug 04 19:44:35.805825 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.805796 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89a8e771-6c979dfbfb-bmgbz"] Aug 04 19:44:35.881452 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.881418 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:35.982661 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:35.982628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:36.081378 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:36.081346 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:36.380723 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:36.380646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:36.600318 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:36.600288 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:37.603431 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:37.603396 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="301bad9c-1f44-4440-a184-655b0c65e947" path="/var/lib/kubelet/pods/301bad9c-1f44-4440-a184-655b0c65e947/volumes" Aug 04 19:44:37.603831 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:37.603789 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c750094-bcab-4d96-be9e-f7aabe3a3202" path="/var/lib/kubelet/pods/6c750094-bcab-4d96-be9e-f7aabe3a3202/volumes" Aug 04 19:44:38.674449 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:38.674363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:38.819294 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:38.819260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:39.320526 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.320485 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:39.424344 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.424309 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:39.461140 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.461110 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp"] Aug 04 19:44:39.461492 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.461453 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301bad9c-1f44-4440-a184-655b0c65e947" containerName="payload-pre-processing" Aug 04 19:44:39.461492 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.461488 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="301bad9c-1f44-4440-a184-655b0c65e947" containerName="payload-pre-processing" Aug 04 19:44:39.461600 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.461507 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c750094-bcab-4d96-be9e-f7aabe3a3202" containerName="payload-processing" Aug 04 19:44:39.461600 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.461513 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c750094-bcab-4d96-be9e-f7aabe3a3202" containerName="payload-processing" Aug 04 19:44:39.461600 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.461576 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="301bad9c-1f44-4440-a184-655b0c65e947" containerName="payload-pre-processing" Aug 04 19:44:39.461600 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.461586 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c750094-bcab-4d96-be9e-f7aabe3a3202" containerName="payload-processing" Aug 04 19:44:39.465909 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.465894 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:44:39.470106 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.470087 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ead82904-dockercfg-qpnwh\"" Aug 04 19:44:39.470219 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.470092 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ead82904\"" Aug 04 19:44:39.479082 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.479061 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp"] Aug 04 19:44:39.516831 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.516800 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt"] Aug 04 19:44:39.520270 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.520247 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:44:39.532717 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.532645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:44:39.534760 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.534735 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt"] Aug 04 19:44:39.619713 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.619637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbptj\" (UniqueName: \"kubernetes.io/projected/5e82f40b-0189-4036-8489-cd103636372b-kube-api-access-hbptj\") pod \"payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp\" (UID: \"5e82f40b-0189-4036-8489-cd103636372b\") " pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:44:39.619713 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.619699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9489855c-cc45-4939-9893-e5882274f9fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt\" (UID: \"9489855c-cc45-4939-9893-e5882274f9fa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:44:39.619883 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.619720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68x4m\" (UniqueName: \"kubernetes.io/projected/9489855c-cc45-4939-9893-e5882274f9fa-kube-api-access-68x4m\") pod \"payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt\" (UID: \"9489855c-cc45-4939-9893-e5882274f9fa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:44:39.619883 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.619777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e82f40b-0189-4036-8489-cd103636372b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp\" (UID: \"5e82f40b-0189-4036-8489-cd103636372b\") " pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:44:39.720766 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.720728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9489855c-cc45-4939-9893-e5882274f9fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt\" (UID: \"9489855c-cc45-4939-9893-e5882274f9fa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:44:39.720766 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.720771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68x4m\" (UniqueName: \"kubernetes.io/projected/9489855c-cc45-4939-9893-e5882274f9fa-kube-api-access-68x4m\") pod \"payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt\" (UID: \"9489855c-cc45-4939-9893-e5882274f9fa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:44:39.721317 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.720859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e82f40b-0189-4036-8489-cd103636372b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp\" (UID: \"5e82f40b-0189-4036-8489-cd103636372b\") " pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:44:39.721317 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.720895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbptj\" (UniqueName: \"kubernetes.io/projected/5e82f40b-0189-4036-8489-cd103636372b-kube-api-access-hbptj\") pod \"payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp\" (UID: \"5e82f40b-0189-4036-8489-cd103636372b\") " pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:44:39.721494 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.721447 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9489855c-cc45-4939-9893-e5882274f9fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt\" (UID: \"9489855c-cc45-4939-9893-e5882274f9fa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:44:39.721567 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.721549 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e82f40b-0189-4036-8489-cd103636372b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp\" (UID: \"5e82f40b-0189-4036-8489-cd103636372b\") " pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:44:39.734247 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.734220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68x4m\" (UniqueName: \"kubernetes.io/projected/9489855c-cc45-4939-9893-e5882274f9fa-kube-api-access-68x4m\") pod \"payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt\" (UID: \"9489855c-cc45-4939-9893-e5882274f9fa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:44:39.734362 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.734223 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbptj\" (UniqueName: \"kubernetes.io/projected/5e82f40b-0189-4036-8489-cd103636372b-kube-api-access-hbptj\") pod \"payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp\" (UID: \"5e82f40b-0189-4036-8489-cd103636372b\") " pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:44:39.776576 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.776546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:44:39.830100 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.830059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:44:39.937180 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.937136 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp"] Aug 04 19:44:39.941920 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:44:39.941880 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e82f40b_0189_4036_8489_cd103636372b.slice/crio-6733e23eb94a05c22a9493fd1776307f23a2ae5386ce8f44f98093ec86f7b177 WatchSource:0}: Error finding container 6733e23eb94a05c22a9493fd1776307f23a2ae5386ce8f44f98093ec86f7b177: Status 404 returned error can't find the container with id 6733e23eb94a05c22a9493fd1776307f23a2ae5386ce8f44f98093ec86f7b177 Aug 04 19:44:39.991577 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:39.991554 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt"] Aug 04 19:44:39.994416 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:44:39.994393 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9489855c_cc45_4939_9893_e5882274f9fa.slice/crio-a1cb3346849d26525cb0c200d4cfd4abe60fc7bdd8d7a1c3f6ab71db8dcb9cbc WatchSource:0}: Error finding container a1cb3346849d26525cb0c200d4cfd4abe60fc7bdd8d7a1c3f6ab71db8dcb9cbc: Status 404 returned error can't find the container with id a1cb3346849d26525cb0c200d4cfd4abe60fc7bdd8d7a1c3f6ab71db8dcb9cbc Aug 04 19:44:40.493803 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:40.493763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" event={"ID":"5e82f40b-0189-4036-8489-cd103636372b","Type":"ContainerStarted","Data":"3ea7b4dee0d4fed10af6a87d7b4ac8d130184331476159233579e2ab8693b806"} Aug 04 19:44:40.494023 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:40.493824 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" event={"ID":"5e82f40b-0189-4036-8489-cd103636372b","Type":"ContainerStarted","Data":"6733e23eb94a05c22a9493fd1776307f23a2ae5386ce8f44f98093ec86f7b177"} Aug 04 19:44:40.494023 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:40.493983 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:44:40.495524 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:40.495499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" event={"ID":"9489855c-cc45-4939-9893-e5882274f9fa","Type":"ContainerStarted","Data":"38c889708578ea952bf6ed5c3b5502f2a1d4c60969a3651cdc1e5efd0745df0f"} Aug 04 19:44:40.495638 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:40.495541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" event={"ID":"9489855c-cc45-4939-9893-e5882274f9fa","Type":"ContainerStarted","Data":"a1cb3346849d26525cb0c200d4cfd4abe60fc7bdd8d7a1c3f6ab71db8dcb9cbc"} Aug 04 19:44:40.495638 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:40.495577 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:44:40.515585 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:40.515522 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" podStartSLOduration=1.515508917 podStartE2EDuration="1.515508917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:44:40.514605662 +0000 UTC m=+1725.420206206" watchObservedRunningTime="2026-08-04 19:44:40.515508917 +0000 UTC m=+1725.421109455" Aug 04 19:44:40.535317 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:40.535268 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" podStartSLOduration=1.535253701 podStartE2EDuration="1.535253701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:44:40.533541338 +0000 UTC m=+1725.439141878" watchObservedRunningTime="2026-08-04 19:44:40.535253701 +0000 UTC m=+1725.440854239" Aug 04 19:44:51.502932 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:51.502902 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:44:51.503293 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:44:51.502982 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:45:02.540825 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:02.540774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:02.596795 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:02.596759 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:09.457948 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.457903 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:09.501981 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.501183 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp"] Aug 04 19:45:09.501981 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.501545 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" podUID="5e82f40b-0189-4036-8489-cd103636372b" containerName="payload-processing" containerID="cri-o://3ea7b4dee0d4fed10af6a87d7b4ac8d130184331476159233579e2ab8693b806" gracePeriod=30 Aug 04 19:45:09.513611 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.513070 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt"] Aug 04 19:45:09.513611 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.513290 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" podUID="9489855c-cc45-4939-9893-e5882274f9fa" containerName="payload-pre-processing" containerID="cri-o://38c889708578ea952bf6ed5c3b5502f2a1d4c60969a3651cdc1e5efd0745df0f" gracePeriod=30 Aug 04 19:45:09.549509 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.549479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:09.606740 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.606710 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:09.634389 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.634358 2580 generic.go:356] "Generic (PLEG): container finished" podID="5e82f40b-0189-4036-8489-cd103636372b" containerID="3ea7b4dee0d4fed10af6a87d7b4ac8d130184331476159233579e2ab8693b806" exitCode=0 Aug 04 19:45:09.634547 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.634403 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" event={"ID":"5e82f40b-0189-4036-8489-cd103636372b","Type":"ContainerDied","Data":"3ea7b4dee0d4fed10af6a87d7b4ac8d130184331476159233579e2ab8693b806"} Aug 04 19:45:09.636226 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.636193 2580 generic.go:356] "Generic (PLEG): container finished" podID="9489855c-cc45-4939-9893-e5882274f9fa" containerID="38c889708578ea952bf6ed5c3b5502f2a1d4c60969a3651cdc1e5efd0745df0f" exitCode=0 Aug 04 19:45:09.636325 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.636246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" event={"ID":"9489855c-cc45-4939-9893-e5882274f9fa","Type":"ContainerDied","Data":"38c889708578ea952bf6ed5c3b5502f2a1d4c60969a3651cdc1e5efd0745df0f"} Aug 04 19:45:09.667210 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.667178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:09.788953 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.788929 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:45:09.791586 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.791567 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:45:09.816705 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.816678 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:09.884338 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.884309 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68x4m\" (UniqueName: \"kubernetes.io/projected/9489855c-cc45-4939-9893-e5882274f9fa-kube-api-access-68x4m\") pod \"9489855c-cc45-4939-9893-e5882274f9fa\" (UID: \"9489855c-cc45-4939-9893-e5882274f9fa\") " Aug 04 19:45:09.884510 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.884367 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e82f40b-0189-4036-8489-cd103636372b-plugins-config-volume\") pod \"5e82f40b-0189-4036-8489-cd103636372b\" (UID: \"5e82f40b-0189-4036-8489-cd103636372b\") " Aug 04 19:45:09.884572 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.884505 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9489855c-cc45-4939-9893-e5882274f9fa-plugins-config-volume\") pod \"9489855c-cc45-4939-9893-e5882274f9fa\" (UID: \"9489855c-cc45-4939-9893-e5882274f9fa\") " Aug 04 19:45:09.884572 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.884542 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbptj\" (UniqueName: \"kubernetes.io/projected/5e82f40b-0189-4036-8489-cd103636372b-kube-api-access-hbptj\") pod \"5e82f40b-0189-4036-8489-cd103636372b\" (UID: \"5e82f40b-0189-4036-8489-cd103636372b\") " Aug 04 19:45:09.884795 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.884768 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e82f40b-0189-4036-8489-cd103636372b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e82f40b-0189-4036-8489-cd103636372b" (UID: "5e82f40b-0189-4036-8489-cd103636372b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:45:09.884891 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.884871 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9489855c-cc45-4939-9893-e5882274f9fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9489855c-cc45-4939-9893-e5882274f9fa" (UID: "9489855c-cc45-4939-9893-e5882274f9fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:45:09.886569 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.886548 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9489855c-cc45-4939-9893-e5882274f9fa-kube-api-access-68x4m" (OuterVolumeSpecName: "kube-api-access-68x4m") pod "9489855c-cc45-4939-9893-e5882274f9fa" (UID: "9489855c-cc45-4939-9893-e5882274f9fa"). InnerVolumeSpecName "kube-api-access-68x4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:45:09.886636 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.886618 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e82f40b-0189-4036-8489-cd103636372b-kube-api-access-hbptj" (OuterVolumeSpecName: "kube-api-access-hbptj") pod "5e82f40b-0189-4036-8489-cd103636372b" (UID: "5e82f40b-0189-4036-8489-cd103636372b"). InnerVolumeSpecName "kube-api-access-hbptj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:45:09.985864 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.985797 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e82f40b-0189-4036-8489-cd103636372b-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:45:09.985864 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.985821 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9489855c-cc45-4939-9893-e5882274f9fa-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:45:09.985864 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.985830 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbptj\" (UniqueName: \"kubernetes.io/projected/5e82f40b-0189-4036-8489-cd103636372b-kube-api-access-hbptj\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:45:09.985864 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:09.985838 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68x4m\" (UniqueName: \"kubernetes.io/projected/9489855c-cc45-4939-9893-e5882274f9fa-kube-api-access-68x4m\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:45:10.025691 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.025670 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:10.247107 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.247033 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:10.597602 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.597564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:10.641737 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.641711 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" Aug 04 19:45:10.641908 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.641738 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp" event={"ID":"5e82f40b-0189-4036-8489-cd103636372b","Type":"ContainerDied","Data":"6733e23eb94a05c22a9493fd1776307f23a2ae5386ce8f44f98093ec86f7b177"} Aug 04 19:45:10.641908 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.641802 2580 scope.go:117] "RemoveContainer" containerID="3ea7b4dee0d4fed10af6a87d7b4ac8d130184331476159233579e2ab8693b806" Aug 04 19:45:10.643390 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.643303 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" event={"ID":"9489855c-cc45-4939-9893-e5882274f9fa","Type":"ContainerDied","Data":"a1cb3346849d26525cb0c200d4cfd4abe60fc7bdd8d7a1c3f6ab71db8dcb9cbc"} Aug 04 19:45:10.643390 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.643328 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt" Aug 04 19:45:10.655241 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.654855 2580 scope.go:117] "RemoveContainer" containerID="38c889708578ea952bf6ed5c3b5502f2a1d4c60969a3651cdc1e5efd0745df0f" Aug 04 19:45:10.675135 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.675103 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp"] Aug 04 19:45:10.687338 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.687313 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ead82904-7f8b45d6b5-tt6qp"] Aug 04 19:45:10.699605 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.699584 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt"] Aug 04 19:45:10.707040 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.707020 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ead82904-69d59c66b8-zcrzt"] Aug 04 19:45:10.878266 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:10.878186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:11.604049 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:11.604022 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e82f40b-0189-4036-8489-cd103636372b" path="/var/lib/kubelet/pods/5e82f40b-0189-4036-8489-cd103636372b/volumes" Aug 04 19:45:11.604396 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:11.604361 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9489855c-cc45-4939-9893-e5882274f9fa" path="/var/lib/kubelet/pods/9489855c-cc45-4939-9893-e5882274f9fa/volumes" Aug 04 19:45:12.713373 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:12.713344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:12.817299 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:12.817268 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:13.830482 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:13.830424 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:14.054848 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.054818 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:14.398957 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.398925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:14.513215 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.513186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:14.623974 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.623945 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx"] Aug 04 19:45:14.624348 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.624335 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9489855c-cc45-4939-9893-e5882274f9fa" containerName="payload-pre-processing" Aug 04 19:45:14.624398 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.624351 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9489855c-cc45-4939-9893-e5882274f9fa" containerName="payload-pre-processing" Aug 04 19:45:14.624398 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.624376 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e82f40b-0189-4036-8489-cd103636372b" containerName="payload-processing" Aug 04 19:45:14.624398 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.624384 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e82f40b-0189-4036-8489-cd103636372b" containerName="payload-processing" Aug 04 19:45:14.624533 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.624442 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9489855c-cc45-4939-9893-e5882274f9fa" containerName="payload-pre-processing" Aug 04 19:45:14.624533 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.624454 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e82f40b-0189-4036-8489-cd103636372b" containerName="payload-processing" Aug 04 19:45:14.629270 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.629255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:14.631896 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.631878 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c53c7015\"" Aug 04 19:45:14.632001 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.631895 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c53c7015-dockercfg-6s2gh\"" Aug 04 19:45:14.638064 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.638044 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx"] Aug 04 19:45:14.688906 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.688839 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v"] Aug 04 19:45:14.692115 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.692099 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:14.701741 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.701719 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v"] Aug 04 19:45:14.725553 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.725520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgh49\" (UniqueName: \"kubernetes.io/projected/a1a2f07a-d196-45e5-91d1-381668eedecf-kube-api-access-jgh49\") pod \"payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx\" (UID: \"a1a2f07a-d196-45e5-91d1-381668eedecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:14.725661 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.725585 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a2f07a-d196-45e5-91d1-381668eedecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx\" (UID: \"a1a2f07a-d196-45e5-91d1-381668eedecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:14.826089 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.826060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ea37b7-dd50-4d70-b4c4-282f772b1592-plugins-config-volume\") pod \"payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v\" (UID: \"28ea37b7-dd50-4d70-b4c4-282f772b1592\") " pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:14.826194 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.826111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9j4p\" (UniqueName: \"kubernetes.io/projected/28ea37b7-dd50-4d70-b4c4-282f772b1592-kube-api-access-f9j4p\") pod \"payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v\" (UID: \"28ea37b7-dd50-4d70-b4c4-282f772b1592\") " pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:14.826194 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.826149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgh49\" (UniqueName: \"kubernetes.io/projected/a1a2f07a-d196-45e5-91d1-381668eedecf-kube-api-access-jgh49\") pod \"payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx\" (UID: \"a1a2f07a-d196-45e5-91d1-381668eedecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:14.826194 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.826174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a2f07a-d196-45e5-91d1-381668eedecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx\" (UID: \"a1a2f07a-d196-45e5-91d1-381668eedecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:14.826811 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.826794 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a2f07a-d196-45e5-91d1-381668eedecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx\" (UID: \"a1a2f07a-d196-45e5-91d1-381668eedecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:14.834833 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.834811 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgh49\" (UniqueName: \"kubernetes.io/projected/a1a2f07a-d196-45e5-91d1-381668eedecf-kube-api-access-jgh49\") pod \"payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx\" (UID: \"a1a2f07a-d196-45e5-91d1-381668eedecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:14.927350 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.927324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ea37b7-dd50-4d70-b4c4-282f772b1592-plugins-config-volume\") pod \"payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v\" (UID: \"28ea37b7-dd50-4d70-b4c4-282f772b1592\") " pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:14.927484 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.927375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9j4p\" (UniqueName: \"kubernetes.io/projected/28ea37b7-dd50-4d70-b4c4-282f772b1592-kube-api-access-f9j4p\") pod \"payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v\" (UID: \"28ea37b7-dd50-4d70-b4c4-282f772b1592\") " pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:14.928033 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.928015 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ea37b7-dd50-4d70-b4c4-282f772b1592-plugins-config-volume\") pod \"payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v\" (UID: \"28ea37b7-dd50-4d70-b4c4-282f772b1592\") " pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:14.936766 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.936744 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9j4p\" (UniqueName: \"kubernetes.io/projected/28ea37b7-dd50-4d70-b4c4-282f772b1592-kube-api-access-f9j4p\") pod \"payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v\" (UID: \"28ea37b7-dd50-4d70-b4c4-282f772b1592\") " pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:14.939617 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:14.939580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:15.003648 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.003619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:15.086563 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.086535 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx"] Aug 04 19:45:15.088739 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:45:15.088712 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a2f07a_d196_45e5_91d1_381668eedecf.slice/crio-5c835ee1b059efc7673c72652d594a165e020db8e2e4e44e451ea10354fad1cb WatchSource:0}: Error finding container 5c835ee1b059efc7673c72652d594a165e020db8e2e4e44e451ea10354fad1cb: Status 404 returned error can't find the container with id 5c835ee1b059efc7673c72652d594a165e020db8e2e4e44e451ea10354fad1cb Aug 04 19:45:15.148375 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.148349 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v"] Aug 04 19:45:15.149948 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:45:15.149927 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ea37b7_dd50_4d70_b4c4_282f772b1592.slice/crio-d8a5a1601b24bbc7e9202c9adb7581c82803ae55f273a5935be739fbb44db183 WatchSource:0}: Error finding container d8a5a1601b24bbc7e9202c9adb7581c82803ae55f273a5935be739fbb44db183: Status 404 returned error can't find the container with id d8a5a1601b24bbc7e9202c9adb7581c82803ae55f273a5935be739fbb44db183 Aug 04 19:45:15.670314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.670220 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" event={"ID":"a1a2f07a-d196-45e5-91d1-381668eedecf","Type":"ContainerStarted","Data":"529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285"} Aug 04 19:45:15.670314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.670289 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:15.670559 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.670317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" event={"ID":"a1a2f07a-d196-45e5-91d1-381668eedecf","Type":"ContainerStarted","Data":"5c835ee1b059efc7673c72652d594a165e020db8e2e4e44e451ea10354fad1cb"} Aug 04 19:45:15.671780 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.671758 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" event={"ID":"28ea37b7-dd50-4d70-b4c4-282f772b1592","Type":"ContainerStarted","Data":"48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8"} Aug 04 19:45:15.671861 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.671807 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:15.671861 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.671823 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" event={"ID":"28ea37b7-dd50-4d70-b4c4-282f772b1592","Type":"ContainerStarted","Data":"d8a5a1601b24bbc7e9202c9adb7581c82803ae55f273a5935be739fbb44db183"} Aug 04 19:45:15.708451 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.708407 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" podStartSLOduration=1.7083926219999999 podStartE2EDuration="1.708392622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:15.707194673 +0000 UTC m=+1760.612795213" watchObservedRunningTime="2026-08-04 19:45:15.708392622 +0000 UTC m=+1760.613993161" Aug 04 19:45:15.709416 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:15.709386 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" podStartSLOduration=1.709373415 podStartE2EDuration="1.709373415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:15.690743948 +0000 UTC m=+1760.596344488" watchObservedRunningTime="2026-08-04 19:45:15.709373415 +0000 UTC m=+1760.614973954" Aug 04 19:45:26.678419 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:26.678389 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:26.678817 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:26.678479 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:29.058188 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:29.058152 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:44.328684 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.328652 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:44.358562 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.358531 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v"] Aug 04 19:45:44.358811 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.358785 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" podUID="28ea37b7-dd50-4d70-b4c4-282f772b1592" containerName="payload-processing" containerID="cri-o://48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8" gracePeriod=30 Aug 04 19:45:44.364179 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.364154 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx"] Aug 04 19:45:44.364517 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.364447 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" podUID="a1a2f07a-d196-45e5-91d1-381668eedecf" containerName="payload-pre-processing" containerID="cri-o://529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285" gracePeriod=30 Aug 04 19:45:44.395600 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.395574 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:44.494259 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.494219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:44.548372 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.548338 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:44.633792 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.633760 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:44.636400 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.636381 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:44.684564 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.684537 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9j4p\" (UniqueName: \"kubernetes.io/projected/28ea37b7-dd50-4d70-b4c4-282f772b1592-kube-api-access-f9j4p\") pod \"28ea37b7-dd50-4d70-b4c4-282f772b1592\" (UID: \"28ea37b7-dd50-4d70-b4c4-282f772b1592\") " Aug 04 19:45:44.684713 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.684641 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgh49\" (UniqueName: \"kubernetes.io/projected/a1a2f07a-d196-45e5-91d1-381668eedecf-kube-api-access-jgh49\") pod \"a1a2f07a-d196-45e5-91d1-381668eedecf\" (UID: \"a1a2f07a-d196-45e5-91d1-381668eedecf\") " Aug 04 19:45:44.684713 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.684674 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a2f07a-d196-45e5-91d1-381668eedecf-plugins-config-volume\") pod \"a1a2f07a-d196-45e5-91d1-381668eedecf\" (UID: \"a1a2f07a-d196-45e5-91d1-381668eedecf\") " Aug 04 19:45:44.684803 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.684745 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ea37b7-dd50-4d70-b4c4-282f772b1592-plugins-config-volume\") pod \"28ea37b7-dd50-4d70-b4c4-282f772b1592\" (UID: \"28ea37b7-dd50-4d70-b4c4-282f772b1592\") " Aug 04 19:45:44.685172 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.685138 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ea37b7-dd50-4d70-b4c4-282f772b1592-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28ea37b7-dd50-4d70-b4c4-282f772b1592" (UID: "28ea37b7-dd50-4d70-b4c4-282f772b1592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:45:44.685307 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.685145 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a2f07a-d196-45e5-91d1-381668eedecf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1a2f07a-d196-45e5-91d1-381668eedecf" (UID: "a1a2f07a-d196-45e5-91d1-381668eedecf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:45:44.686738 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.686715 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a2f07a-d196-45e5-91d1-381668eedecf-kube-api-access-jgh49" (OuterVolumeSpecName: "kube-api-access-jgh49") pod "a1a2f07a-d196-45e5-91d1-381668eedecf" (UID: "a1a2f07a-d196-45e5-91d1-381668eedecf"). InnerVolumeSpecName "kube-api-access-jgh49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:45:44.686852 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.686778 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ea37b7-dd50-4d70-b4c4-282f772b1592-kube-api-access-f9j4p" (OuterVolumeSpecName: "kube-api-access-f9j4p") pod "28ea37b7-dd50-4d70-b4c4-282f772b1592" (UID: "28ea37b7-dd50-4d70-b4c4-282f772b1592"). InnerVolumeSpecName "kube-api-access-f9j4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:45:44.785830 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.785801 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ea37b7-dd50-4d70-b4c4-282f772b1592-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:45:44.785830 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.785825 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9j4p\" (UniqueName: \"kubernetes.io/projected/28ea37b7-dd50-4d70-b4c4-282f772b1592-kube-api-access-f9j4p\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:45:44.785830 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.785834 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgh49\" (UniqueName: \"kubernetes.io/projected/a1a2f07a-d196-45e5-91d1-381668eedecf-kube-api-access-jgh49\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:45:44.786046 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.785844 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a2f07a-d196-45e5-91d1-381668eedecf-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:45:44.807189 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.807161 2580 generic.go:356] "Generic (PLEG): container finished" podID="28ea37b7-dd50-4d70-b4c4-282f772b1592" containerID="48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8" exitCode=0 Aug 04 19:45:44.807322 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.807209 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" Aug 04 19:45:44.807322 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.807249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" event={"ID":"28ea37b7-dd50-4d70-b4c4-282f772b1592","Type":"ContainerDied","Data":"48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8"} Aug 04 19:45:44.807322 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.807299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v" event={"ID":"28ea37b7-dd50-4d70-b4c4-282f772b1592","Type":"ContainerDied","Data":"d8a5a1601b24bbc7e9202c9adb7581c82803ae55f273a5935be739fbb44db183"} Aug 04 19:45:44.807545 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.807329 2580 scope.go:117] "RemoveContainer" containerID="48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8" Aug 04 19:45:44.808663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.808640 2580 generic.go:356] "Generic (PLEG): container finished" podID="a1a2f07a-d196-45e5-91d1-381668eedecf" containerID="529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285" exitCode=0 Aug 04 19:45:44.808766 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.808692 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" Aug 04 19:45:44.808766 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.808703 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" event={"ID":"a1a2f07a-d196-45e5-91d1-381668eedecf","Type":"ContainerDied","Data":"529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285"} Aug 04 19:45:44.808766 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.808735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx" event={"ID":"a1a2f07a-d196-45e5-91d1-381668eedecf","Type":"ContainerDied","Data":"5c835ee1b059efc7673c72652d594a165e020db8e2e4e44e451ea10354fad1cb"} Aug 04 19:45:44.817708 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.817693 2580 scope.go:117] "RemoveContainer" containerID="48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8" Aug 04 19:45:44.817987 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:45:44.817967 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8\": container with ID starting with 48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8 not found: ID does not exist" containerID="48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8" Aug 04 19:45:44.818084 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.817999 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8"} err="failed to get container status \"48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8\": rpc error: code = NotFound desc = could not find container \"48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8\": container with ID starting with 48266e87bd6b7bba1907621071bc61314b9a9e2bda997fb2860760795496afe8 not found: ID does not exist" Aug 04 19:45:44.818084 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.818034 2580 scope.go:117] "RemoveContainer" containerID="529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285" Aug 04 19:45:44.847267 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.847249 2580 scope.go:117] "RemoveContainer" containerID="529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285" Aug 04 19:45:44.847582 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:45:44.847561 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285\": container with ID starting with 529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285 not found: ID does not exist" containerID="529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285" Aug 04 19:45:44.847636 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.847591 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285"} err="failed to get container status \"529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285\": rpc error: code = NotFound desc = could not find container \"529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285\": container with ID starting with 529383b79f212e24f08bed0c93de58fe5a4a4b55ce1cc82771b1a688b0bf9285 not found: ID does not exist" Aug 04 19:45:44.864045 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.863998 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v"] Aug 04 19:45:44.875942 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.875919 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c53c7015-654fb784bb-5h99v"] Aug 04 19:45:44.910749 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.910728 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx"] Aug 04 19:45:44.912247 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:44.912226 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c53c7015-6d8969956-k58dx"] Aug 04 19:45:45.603489 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:45.603433 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ea37b7-dd50-4d70-b4c4-282f772b1592" path="/var/lib/kubelet/pods/28ea37b7-dd50-4d70-b4c4-282f772b1592/volumes" Aug 04 19:45:45.603881 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:45.603832 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a2f07a-d196-45e5-91d1-381668eedecf" path="/var/lib/kubelet/pods/a1a2f07a-d196-45e5-91d1-381668eedecf/volumes" Aug 04 19:45:45.708899 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:45.708859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:45.914286 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:45.914252 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:46.125210 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:46.125099 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:46.417782 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:46.417685 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:47.335106 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:47.335080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:48.926754 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:48.926717 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:49.019431 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:49.019383 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:50.151966 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:50.151924 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:50.372966 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:50.372935 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:50.803845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:50.803805 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:50.908271 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:50.908238 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:45:51.050160 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.050123 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27"] Aug 04 19:45:51.050520 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.050508 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a2f07a-d196-45e5-91d1-381668eedecf" containerName="payload-pre-processing" Aug 04 19:45:51.050574 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.050523 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a2f07a-d196-45e5-91d1-381668eedecf" containerName="payload-pre-processing" Aug 04 19:45:51.050574 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.050535 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ea37b7-dd50-4d70-b4c4-282f772b1592" containerName="payload-processing" Aug 04 19:45:51.050574 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.050540 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ea37b7-dd50-4d70-b4c4-282f772b1592" containerName="payload-processing" Aug 04 19:45:51.050723 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.050596 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ea37b7-dd50-4d70-b4c4-282f772b1592" containerName="payload-processing" Aug 04 19:45:51.050723 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.050606 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a2f07a-d196-45e5-91d1-381668eedecf" containerName="payload-pre-processing" Aug 04 19:45:51.055206 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.055154 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:45:51.058012 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.057988 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3d6cfa2c-dockercfg-5d5bt\"" Aug 04 19:45:51.058450 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.058432 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3d6cfa2c\"" Aug 04 19:45:51.065325 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.065305 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27"] Aug 04 19:45:51.104417 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.104385 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz"] Aug 04 19:45:51.107839 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.107825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:45:51.124672 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.124646 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz"] Aug 04 19:45:51.146427 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.146399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv755\" (UniqueName: \"kubernetes.io/projected/36880a8d-88a4-47cf-9e80-c1be6db1f73b-kube-api-access-gv755\") pod \"payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz\" (UID: \"36880a8d-88a4-47cf-9e80-c1be6db1f73b\") " pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:45:51.146593 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.146434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934fb374-fc99-48ca-9dba-f8201c840425-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27\" (UID: \"934fb374-fc99-48ca-9dba-f8201c840425\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:45:51.146593 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.146451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4qr\" (UniqueName: \"kubernetes.io/projected/934fb374-fc99-48ca-9dba-f8201c840425-kube-api-access-6t4qr\") pod \"payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27\" (UID: \"934fb374-fc99-48ca-9dba-f8201c840425\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:45:51.146723 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.146587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36880a8d-88a4-47cf-9e80-c1be6db1f73b-plugins-config-volume\") pod \"payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz\" (UID: \"36880a8d-88a4-47cf-9e80-c1be6db1f73b\") " pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:45:51.247740 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.247702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv755\" (UniqueName: \"kubernetes.io/projected/36880a8d-88a4-47cf-9e80-c1be6db1f73b-kube-api-access-gv755\") pod \"payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz\" (UID: \"36880a8d-88a4-47cf-9e80-c1be6db1f73b\") " pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:45:51.247740 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.247744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934fb374-fc99-48ca-9dba-f8201c840425-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27\" (UID: \"934fb374-fc99-48ca-9dba-f8201c840425\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:45:51.248249 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.247765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4qr\" (UniqueName: \"kubernetes.io/projected/934fb374-fc99-48ca-9dba-f8201c840425-kube-api-access-6t4qr\") pod \"payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27\" (UID: \"934fb374-fc99-48ca-9dba-f8201c840425\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:45:51.248249 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.247829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36880a8d-88a4-47cf-9e80-c1be6db1f73b-plugins-config-volume\") pod \"payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz\" (UID: \"36880a8d-88a4-47cf-9e80-c1be6db1f73b\") " pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:45:51.248548 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.248524 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934fb374-fc99-48ca-9dba-f8201c840425-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27\" (UID: \"934fb374-fc99-48ca-9dba-f8201c840425\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:45:51.248748 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.248724 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36880a8d-88a4-47cf-9e80-c1be6db1f73b-plugins-config-volume\") pod \"payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz\" (UID: \"36880a8d-88a4-47cf-9e80-c1be6db1f73b\") " pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:45:51.256379 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.256354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv755\" (UniqueName: \"kubernetes.io/projected/36880a8d-88a4-47cf-9e80-c1be6db1f73b-kube-api-access-gv755\") pod \"payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz\" (UID: \"36880a8d-88a4-47cf-9e80-c1be6db1f73b\") " pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:45:51.256495 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.256423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4qr\" (UniqueName: \"kubernetes.io/projected/934fb374-fc99-48ca-9dba-f8201c840425-kube-api-access-6t4qr\") pod \"payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27\" (UID: \"934fb374-fc99-48ca-9dba-f8201c840425\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:45:51.365984 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.365894 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:45:51.417910 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.417870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:45:51.511291 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.511267 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27"] Aug 04 19:45:51.514723 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:45:51.514698 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934fb374_fc99_48ca_9dba_f8201c840425.slice/crio-6cd68d340056aa75ac981e0b09587ca26e50e8e19fbd3e9afa7e405d7b27a4c2 WatchSource:0}: Error finding container 6cd68d340056aa75ac981e0b09587ca26e50e8e19fbd3e9afa7e405d7b27a4c2: Status 404 returned error can't find the container with id 6cd68d340056aa75ac981e0b09587ca26e50e8e19fbd3e9afa7e405d7b27a4c2 Aug 04 19:45:51.579713 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.579598 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz"] Aug 04 19:45:51.581173 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:45:51.581149 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36880a8d_88a4_47cf_9e80_c1be6db1f73b.slice/crio-fb47eed146ba444327d05852970e9f5c6383d9ead76f43c4dd9cb1790df7d543 WatchSource:0}: Error finding container fb47eed146ba444327d05852970e9f5c6383d9ead76f43c4dd9cb1790df7d543: Status 404 returned error can't find the container with id fb47eed146ba444327d05852970e9f5c6383d9ead76f43c4dd9cb1790df7d543 Aug 04 19:45:51.846032 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.845997 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" event={"ID":"36880a8d-88a4-47cf-9e80-c1be6db1f73b","Type":"ContainerStarted","Data":"d681e46e9cd279b213f2651193d681aff10dde13dfefe6cdbc4ea4d97b9666f3"} Aug 04 19:45:51.846230 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.846058 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" event={"ID":"36880a8d-88a4-47cf-9e80-c1be6db1f73b","Type":"ContainerStarted","Data":"fb47eed146ba444327d05852970e9f5c6383d9ead76f43c4dd9cb1790df7d543"} Aug 04 19:45:51.846230 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.846174 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:45:51.847517 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.847494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" event={"ID":"934fb374-fc99-48ca-9dba-f8201c840425","Type":"ContainerStarted","Data":"e7b374ee99e380b514b1f2b71c3b8c07bb8aff86f60b473b3b36eb08a18479dd"} Aug 04 19:45:51.847632 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.847531 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" event={"ID":"934fb374-fc99-48ca-9dba-f8201c840425","Type":"ContainerStarted","Data":"6cd68d340056aa75ac981e0b09587ca26e50e8e19fbd3e9afa7e405d7b27a4c2"} Aug 04 19:45:51.847632 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.847614 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:45:51.863148 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.863102 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" podStartSLOduration=0.863090029 podStartE2EDuration="863.090029ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:51.861846176 +0000 UTC m=+1796.767446715" watchObservedRunningTime="2026-08-04 19:45:51.863090029 +0000 UTC m=+1796.768690545" Aug 04 19:45:51.881060 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:51.880967 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" podStartSLOduration=0.880947637 podStartE2EDuration="880.947637ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:51.880833105 +0000 UTC m=+1796.786433644" watchObservedRunningTime="2026-08-04 19:45:51.880947637 +0000 UTC m=+1796.786548177" Aug 04 19:45:55.607621 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:55.607596 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:45:55.611083 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:45:55.611065 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:46:02.855100 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:02.855066 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:46:02.855593 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:02.855231 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:46:20.714502 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.714443 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:20.751748 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.751716 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz"] Aug 04 19:46:20.752206 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.752173 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" podUID="36880a8d-88a4-47cf-9e80-c1be6db1f73b" containerName="payload-processing" containerID="cri-o://d681e46e9cd279b213f2651193d681aff10dde13dfefe6cdbc4ea4d97b9666f3" gracePeriod=30 Aug 04 19:46:20.756126 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.756090 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27"] Aug 04 19:46:20.756829 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.756385 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" podUID="934fb374-fc99-48ca-9dba-f8201c840425" containerName="payload-pre-processing" containerID="cri-o://e7b374ee99e380b514b1f2b71c3b8c07bb8aff86f60b473b3b36eb08a18479dd" gracePeriod=30 Aug 04 19:46:20.790124 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.790095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:20.849416 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.849389 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:20.950240 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.950207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:20.982808 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.982720 2580 generic.go:356] "Generic (PLEG): container finished" podID="934fb374-fc99-48ca-9dba-f8201c840425" containerID="e7b374ee99e380b514b1f2b71c3b8c07bb8aff86f60b473b3b36eb08a18479dd" exitCode=0 Aug 04 19:46:20.982958 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.982853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" event={"ID":"934fb374-fc99-48ca-9dba-f8201c840425","Type":"ContainerDied","Data":"e7b374ee99e380b514b1f2b71c3b8c07bb8aff86f60b473b3b36eb08a18479dd"} Aug 04 19:46:20.984937 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.984913 2580 generic.go:356] "Generic (PLEG): container finished" podID="36880a8d-88a4-47cf-9e80-c1be6db1f73b" containerID="d681e46e9cd279b213f2651193d681aff10dde13dfefe6cdbc4ea4d97b9666f3" exitCode=0 Aug 04 19:46:20.985080 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:20.984969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" event={"ID":"36880a8d-88a4-47cf-9e80-c1be6db1f73b","Type":"ContainerDied","Data":"d681e46e9cd279b213f2651193d681aff10dde13dfefe6cdbc4ea4d97b9666f3"} Aug 04 19:46:21.039145 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.039117 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:46:21.041266 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.041248 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:46:21.051719 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.051694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:21.111658 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.111635 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36880a8d-88a4-47cf-9e80-c1be6db1f73b-plugins-config-volume\") pod \"36880a8d-88a4-47cf-9e80-c1be6db1f73b\" (UID: \"36880a8d-88a4-47cf-9e80-c1be6db1f73b\") " Aug 04 19:46:21.111781 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.111674 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv755\" (UniqueName: \"kubernetes.io/projected/36880a8d-88a4-47cf-9e80-c1be6db1f73b-kube-api-access-gv755\") pod \"36880a8d-88a4-47cf-9e80-c1be6db1f73b\" (UID: \"36880a8d-88a4-47cf-9e80-c1be6db1f73b\") " Aug 04 19:46:21.111781 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.111732 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934fb374-fc99-48ca-9dba-f8201c840425-plugins-config-volume\") pod \"934fb374-fc99-48ca-9dba-f8201c840425\" (UID: \"934fb374-fc99-48ca-9dba-f8201c840425\") " Aug 04 19:46:21.111781 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.111755 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t4qr\" (UniqueName: \"kubernetes.io/projected/934fb374-fc99-48ca-9dba-f8201c840425-kube-api-access-6t4qr\") pod \"934fb374-fc99-48ca-9dba-f8201c840425\" (UID: \"934fb374-fc99-48ca-9dba-f8201c840425\") " Aug 04 19:46:21.112060 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.112032 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36880a8d-88a4-47cf-9e80-c1be6db1f73b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36880a8d-88a4-47cf-9e80-c1be6db1f73b" (UID: "36880a8d-88a4-47cf-9e80-c1be6db1f73b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:46:21.112113 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.112097 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934fb374-fc99-48ca-9dba-f8201c840425-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "934fb374-fc99-48ca-9dba-f8201c840425" (UID: "934fb374-fc99-48ca-9dba-f8201c840425"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:46:21.113991 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.113966 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36880a8d-88a4-47cf-9e80-c1be6db1f73b-kube-api-access-gv755" (OuterVolumeSpecName: "kube-api-access-gv755") pod "36880a8d-88a4-47cf-9e80-c1be6db1f73b" (UID: "36880a8d-88a4-47cf-9e80-c1be6db1f73b"). InnerVolumeSpecName "kube-api-access-gv755". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:46:21.114086 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.114007 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934fb374-fc99-48ca-9dba-f8201c840425-kube-api-access-6t4qr" (OuterVolumeSpecName: "kube-api-access-6t4qr") pod "934fb374-fc99-48ca-9dba-f8201c840425" (UID: "934fb374-fc99-48ca-9dba-f8201c840425"). InnerVolumeSpecName "kube-api-access-6t4qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:46:21.212431 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.212405 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934fb374-fc99-48ca-9dba-f8201c840425-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:46:21.212431 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.212427 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t4qr\" (UniqueName: \"kubernetes.io/projected/934fb374-fc99-48ca-9dba-f8201c840425-kube-api-access-6t4qr\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:46:21.212589 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.212436 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36880a8d-88a4-47cf-9e80-c1be6db1f73b-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:46:21.212589 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.212447 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv755\" (UniqueName: \"kubernetes.io/projected/36880a8d-88a4-47cf-9e80-c1be6db1f73b-kube-api-access-gv755\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:46:21.287344 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.287265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:21.804262 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.804225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:21.890264 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.890191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:21.991256 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.991224 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:21.991790 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.991775 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" Aug 04 19:46:21.991880 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.991817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27" event={"ID":"934fb374-fc99-48ca-9dba-f8201c840425","Type":"ContainerDied","Data":"6cd68d340056aa75ac981e0b09587ca26e50e8e19fbd3e9afa7e405d7b27a4c2"} Aug 04 19:46:21.991951 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.991879 2580 scope.go:117] "RemoveContainer" containerID="e7b374ee99e380b514b1f2b71c3b8c07bb8aff86f60b473b3b36eb08a18479dd" Aug 04 19:46:21.993322 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.993166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" event={"ID":"36880a8d-88a4-47cf-9e80-c1be6db1f73b","Type":"ContainerDied","Data":"fb47eed146ba444327d05852970e9f5c6383d9ead76f43c4dd9cb1790df7d543"} Aug 04 19:46:21.993322 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:21.993239 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz" Aug 04 19:46:22.002417 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:22.002400 2580 scope.go:117] "RemoveContainer" containerID="d681e46e9cd279b213f2651193d681aff10dde13dfefe6cdbc4ea4d97b9666f3" Aug 04 19:46:22.013061 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:22.013039 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz"] Aug 04 19:46:22.016581 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:22.016562 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3d6cfa2c-5dbdfbf7f4-ddtwz"] Aug 04 19:46:22.036482 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:22.033989 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27"] Aug 04 19:46:22.039718 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:22.039695 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3d6cfa2c-848cc6695b-84h27"] Aug 04 19:46:23.603806 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:23.603775 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36880a8d-88a4-47cf-9e80-c1be6db1f73b" path="/var/lib/kubelet/pods/36880a8d-88a4-47cf-9e80-c1be6db1f73b/volumes" Aug 04 19:46:23.604172 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:23.604116 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934fb374-fc99-48ca-9dba-f8201c840425" path="/var/lib/kubelet/pods/934fb374-fc99-48ca-9dba-f8201c840425/volumes" Aug 04 19:46:33.656887 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:33.656854 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:40.060725 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:40.060683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:40.165582 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:40.165532 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:40.284069 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:40.284034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:40.377391 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:40.377296 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:46:40.814139 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:46:40.814103 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:06.830356 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:06.830310 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:12.706002 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:12.705961 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:12.765167 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:12.765106 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:12.874831 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:12.874752 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:12.970312 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:12.970226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:28.874209 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:28.874106 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:34.494837 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:34.494798 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:34.552101 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:34.552072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:34.629628 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:34.629604 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:34.735606 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:34.735576 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:45.189931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:45.189875 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:47:56.220010 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:47:56.219975 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:02.424234 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:02.424200 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:04.015146 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:04.015112 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:05.093998 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:05.093960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:11.992846 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:11.992801 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:12.094985 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:12.094948 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:12.686730 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:12.686697 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:19.033698 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:19.033660 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:19.134059 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:19.133999 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:19.259576 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:19.259534 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:19.581274 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:19.581213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:19.675046 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:19.675006 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:22.515651 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:22.515609 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:22.627831 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:22.627792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:23.783084 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:23.783049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:24.257590 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.257553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:24.349854 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.349819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:24.358191 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.358160 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27"] Aug 04 19:48:24.358548 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.358535 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934fb374-fc99-48ca-9dba-f8201c840425" containerName="payload-pre-processing" Aug 04 19:48:24.358607 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.358550 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="934fb374-fc99-48ca-9dba-f8201c840425" containerName="payload-pre-processing" Aug 04 19:48:24.358607 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.358575 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36880a8d-88a4-47cf-9e80-c1be6db1f73b" containerName="payload-processing" Aug 04 19:48:24.358607 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.358581 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="36880a8d-88a4-47cf-9e80-c1be6db1f73b" containerName="payload-processing" Aug 04 19:48:24.358717 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.358637 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="934fb374-fc99-48ca-9dba-f8201c840425" containerName="payload-pre-processing" Aug 04 19:48:24.358717 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.358644 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="36880a8d-88a4-47cf-9e80-c1be6db1f73b" containerName="payload-processing" Aug 04 19:48:24.361515 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.361500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:48:24.364786 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.364764 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8129f458-dockercfg-r5qqx\"" Aug 04 19:48:24.364912 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.364898 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8129f458\"" Aug 04 19:48:24.374180 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.374154 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27"] Aug 04 19:48:24.376134 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.376116 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r"] Aug 04 19:48:24.379673 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.379652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:48:24.390208 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.390182 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r"] Aug 04 19:48:24.460570 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.460540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e935cf-361a-4e4c-b680-bc467c960f65-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27\" (UID: \"07e935cf-361a-4e4c-b680-bc467c960f65\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:48:24.460701 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.460641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl5ch\" (UniqueName: \"kubernetes.io/projected/07e935cf-361a-4e4c-b680-bc467c960f65-kube-api-access-vl5ch\") pod \"payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27\" (UID: \"07e935cf-361a-4e4c-b680-bc467c960f65\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:48:24.460751 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.460728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b67391-5b70-4ff1-947c-89d6cacc78dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r\" (UID: \"08b67391-5b70-4ff1-947c-89d6cacc78dd\") " pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:48:24.460799 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.460783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swq9\" (UniqueName: \"kubernetes.io/projected/08b67391-5b70-4ff1-947c-89d6cacc78dd-kube-api-access-9swq9\") pod \"payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r\" (UID: \"08b67391-5b70-4ff1-947c-89d6cacc78dd\") " pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:48:24.491866 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.491832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:24.561346 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.561267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5ch\" (UniqueName: \"kubernetes.io/projected/07e935cf-361a-4e4c-b680-bc467c960f65-kube-api-access-vl5ch\") pod \"payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27\" (UID: \"07e935cf-361a-4e4c-b680-bc467c960f65\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:48:24.561346 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.561310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b67391-5b70-4ff1-947c-89d6cacc78dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r\" (UID: \"08b67391-5b70-4ff1-947c-89d6cacc78dd\") " pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:48:24.561571 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.561353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swq9\" (UniqueName: \"kubernetes.io/projected/08b67391-5b70-4ff1-947c-89d6cacc78dd-kube-api-access-9swq9\") pod \"payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r\" (UID: \"08b67391-5b70-4ff1-947c-89d6cacc78dd\") " pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:48:24.561571 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.561382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e935cf-361a-4e4c-b680-bc467c960f65-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27\" (UID: \"07e935cf-361a-4e4c-b680-bc467c960f65\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:48:24.562110 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.562087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e935cf-361a-4e4c-b680-bc467c960f65-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27\" (UID: \"07e935cf-361a-4e4c-b680-bc467c960f65\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:48:24.562110 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.562104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b67391-5b70-4ff1-947c-89d6cacc78dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r\" (UID: \"08b67391-5b70-4ff1-947c-89d6cacc78dd\") " pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:48:24.569216 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.569182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swq9\" (UniqueName: \"kubernetes.io/projected/08b67391-5b70-4ff1-947c-89d6cacc78dd-kube-api-access-9swq9\") pod \"payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r\" (UID: \"08b67391-5b70-4ff1-947c-89d6cacc78dd\") " pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:48:24.569322 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.569293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5ch\" (UniqueName: \"kubernetes.io/projected/07e935cf-361a-4e4c-b680-bc467c960f65-kube-api-access-vl5ch\") pod \"payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27\" (UID: \"07e935cf-361a-4e4c-b680-bc467c960f65\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:48:24.672441 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.672395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:48:24.690603 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.690579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:48:24.817548 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:24.817519 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27"] Aug 04 19:48:24.819399 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:48:24.819334 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e935cf_361a_4e4c_b680_bc467c960f65.slice/crio-baaf0ab1e4c9408525aec77e5e8cdfc4a9e8ea57a49cf55ca60743e6b52c02fd WatchSource:0}: Error finding container baaf0ab1e4c9408525aec77e5e8cdfc4a9e8ea57a49cf55ca60743e6b52c02fd: Status 404 returned error can't find the container with id baaf0ab1e4c9408525aec77e5e8cdfc4a9e8ea57a49cf55ca60743e6b52c02fd Aug 04 19:48:25.048022 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:25.047996 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r"] Aug 04 19:48:25.049975 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:48:25.049946 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b67391_5b70_4ff1_947c_89d6cacc78dd.slice/crio-a7c11a7ea91c595307fd4f8ec9ee26229718b3f9fb837f0db7e55b7710bc38f3 WatchSource:0}: Error finding container a7c11a7ea91c595307fd4f8ec9ee26229718b3f9fb837f0db7e55b7710bc38f3: Status 404 returned error can't find the container with id a7c11a7ea91c595307fd4f8ec9ee26229718b3f9fb837f0db7e55b7710bc38f3 Aug 04 19:48:25.568744 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:25.568702 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" event={"ID":"08b67391-5b70-4ff1-947c-89d6cacc78dd","Type":"ContainerStarted","Data":"94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed"} Aug 04 19:48:25.568931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:25.568763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" event={"ID":"08b67391-5b70-4ff1-947c-89d6cacc78dd","Type":"ContainerStarted","Data":"a7c11a7ea91c595307fd4f8ec9ee26229718b3f9fb837f0db7e55b7710bc38f3"} Aug 04 19:48:25.568931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:25.568793 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:48:25.570515 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:25.570485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" event={"ID":"07e935cf-361a-4e4c-b680-bc467c960f65","Type":"ContainerStarted","Data":"c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6"} Aug 04 19:48:25.570648 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:25.570537 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" event={"ID":"07e935cf-361a-4e4c-b680-bc467c960f65","Type":"ContainerStarted","Data":"baaf0ab1e4c9408525aec77e5e8cdfc4a9e8ea57a49cf55ca60743e6b52c02fd"} Aug 04 19:48:25.570648 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:25.570569 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:48:25.584583 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:25.584545 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" podStartSLOduration=1.584531771 podStartE2EDuration="1.584531771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:48:25.58262751 +0000 UTC m=+1950.488228050" watchObservedRunningTime="2026-08-04 19:48:25.584531771 +0000 UTC m=+1950.490132309" Aug 04 19:48:25.603262 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:25.599306 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" podStartSLOduration=1.599280306 podStartE2EDuration="1.599280306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:48:25.596637921 +0000 UTC m=+1950.502238453" watchObservedRunningTime="2026-08-04 19:48:25.599280306 +0000 UTC m=+1950.504880845" Aug 04 19:48:25.852325 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:25.852249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:36.577671 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:36.577638 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:48:36.578248 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:36.577779 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:48:47.485173 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:47.485090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:48.201298 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:48.201269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:48.301120 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:48.301093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:48.658316 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:48.658221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:54.317018 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:54.316981 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:48:54.376783 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:48:54.376746 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:05.235846 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.235809 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:05.259997 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.259968 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r"] Aug 04 19:49:05.260295 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.260268 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" podUID="08b67391-5b70-4ff1-947c-89d6cacc78dd" containerName="payload-processing" containerID="cri-o://94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed" gracePeriod=30 Aug 04 19:49:05.266529 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.266502 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27"] Aug 04 19:49:05.266778 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.266729 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" podUID="07e935cf-361a-4e4c-b680-bc467c960f65" containerName="payload-pre-processing" containerID="cri-o://c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6" gracePeriod=30 Aug 04 19:49:05.297078 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.297054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:05.349812 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.349787 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:05.403601 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.403569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:05.515146 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.515125 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:49:05.544272 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.544253 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:49:05.616108 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.616083 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b67391-5b70-4ff1-947c-89d6cacc78dd-plugins-config-volume\") pod \"08b67391-5b70-4ff1-947c-89d6cacc78dd\" (UID: \"08b67391-5b70-4ff1-947c-89d6cacc78dd\") " Aug 04 19:49:05.616230 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.616135 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e935cf-361a-4e4c-b680-bc467c960f65-plugins-config-volume\") pod \"07e935cf-361a-4e4c-b680-bc467c960f65\" (UID: \"07e935cf-361a-4e4c-b680-bc467c960f65\") " Aug 04 19:49:05.616271 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.616239 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl5ch\" (UniqueName: \"kubernetes.io/projected/07e935cf-361a-4e4c-b680-bc467c960f65-kube-api-access-vl5ch\") pod \"07e935cf-361a-4e4c-b680-bc467c960f65\" (UID: \"07e935cf-361a-4e4c-b680-bc467c960f65\") " Aug 04 19:49:05.616271 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.616265 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9swq9\" (UniqueName: \"kubernetes.io/projected/08b67391-5b70-4ff1-947c-89d6cacc78dd-kube-api-access-9swq9\") pod \"08b67391-5b70-4ff1-947c-89d6cacc78dd\" (UID: \"08b67391-5b70-4ff1-947c-89d6cacc78dd\") " Aug 04 19:49:05.616548 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.616526 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b67391-5b70-4ff1-947c-89d6cacc78dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08b67391-5b70-4ff1-947c-89d6cacc78dd" (UID: "08b67391-5b70-4ff1-947c-89d6cacc78dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:49:05.616597 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.616526 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e935cf-361a-4e4c-b680-bc467c960f65-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07e935cf-361a-4e4c-b680-bc467c960f65" (UID: "07e935cf-361a-4e4c-b680-bc467c960f65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:49:05.618433 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.618406 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b67391-5b70-4ff1-947c-89d6cacc78dd-kube-api-access-9swq9" (OuterVolumeSpecName: "kube-api-access-9swq9") pod "08b67391-5b70-4ff1-947c-89d6cacc78dd" (UID: "08b67391-5b70-4ff1-947c-89d6cacc78dd"). InnerVolumeSpecName "kube-api-access-9swq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:49:05.618548 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.618449 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e935cf-361a-4e4c-b680-bc467c960f65-kube-api-access-vl5ch" (OuterVolumeSpecName: "kube-api-access-vl5ch") pod "07e935cf-361a-4e4c-b680-bc467c960f65" (UID: "07e935cf-361a-4e4c-b680-bc467c960f65"). InnerVolumeSpecName "kube-api-access-vl5ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:49:05.717885 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.717851 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b67391-5b70-4ff1-947c-89d6cacc78dd-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:49:05.717885 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.717879 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e935cf-361a-4e4c-b680-bc467c960f65-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:49:05.717885 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.717890 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl5ch\" (UniqueName: \"kubernetes.io/projected/07e935cf-361a-4e4c-b680-bc467c960f65-kube-api-access-vl5ch\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:49:05.718095 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.717898 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9swq9\" (UniqueName: \"kubernetes.io/projected/08b67391-5b70-4ff1-947c-89d6cacc78dd-kube-api-access-9swq9\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:49:05.754227 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.754154 2580 generic.go:356] "Generic (PLEG): container finished" podID="08b67391-5b70-4ff1-947c-89d6cacc78dd" containerID="94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed" exitCode=0 Aug 04 19:49:05.754227 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.754202 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" Aug 04 19:49:05.754424 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.754247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" event={"ID":"08b67391-5b70-4ff1-947c-89d6cacc78dd","Type":"ContainerDied","Data":"94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed"} Aug 04 19:49:05.754424 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.754305 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r" event={"ID":"08b67391-5b70-4ff1-947c-89d6cacc78dd","Type":"ContainerDied","Data":"a7c11a7ea91c595307fd4f8ec9ee26229718b3f9fb837f0db7e55b7710bc38f3"} Aug 04 19:49:05.754424 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.754331 2580 scope.go:117] "RemoveContainer" containerID="94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed" Aug 04 19:49:05.755553 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.755531 2580 generic.go:356] "Generic (PLEG): container finished" podID="07e935cf-361a-4e4c-b680-bc467c960f65" containerID="c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6" exitCode=0 Aug 04 19:49:05.755639 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.755592 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" Aug 04 19:49:05.755696 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.755590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" event={"ID":"07e935cf-361a-4e4c-b680-bc467c960f65","Type":"ContainerDied","Data":"c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6"} Aug 04 19:49:05.755750 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.755732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27" event={"ID":"07e935cf-361a-4e4c-b680-bc467c960f65","Type":"ContainerDied","Data":"baaf0ab1e4c9408525aec77e5e8cdfc4a9e8ea57a49cf55ca60743e6b52c02fd"} Aug 04 19:49:05.763622 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.763606 2580 scope.go:117] "RemoveContainer" containerID="94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed" Aug 04 19:49:05.763920 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:49:05.763897 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed\": container with ID starting with 94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed not found: ID does not exist" containerID="94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed" Aug 04 19:49:05.764000 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.763935 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed"} err="failed to get container status \"94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed\": rpc error: code = NotFound desc = could not find container \"94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed\": container with ID starting with 94152f5d326b7c23b099a53ad8726a576fb17b117504c271e0edc172ee74f7ed not found: ID does not exist" Aug 04 19:49:05.764000 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.763957 2580 scope.go:117] "RemoveContainer" containerID="c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6" Aug 04 19:49:05.771976 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.771956 2580 scope.go:117] "RemoveContainer" containerID="c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6" Aug 04 19:49:05.772236 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:49:05.772211 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6\": container with ID starting with c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6 not found: ID does not exist" containerID="c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6" Aug 04 19:49:05.772310 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.772242 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6"} err="failed to get container status \"c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6\": rpc error: code = NotFound desc = could not find container \"c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6\": container with ID starting with c7fd1af239f863f906ec0667be1d5e86445bce870d7280445c1adb30a83924a6 not found: ID does not exist" Aug 04 19:49:05.778804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.778779 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r"] Aug 04 19:49:05.780688 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.780669 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8129f458-759f8c9f98-5ht8r"] Aug 04 19:49:05.791529 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.791509 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27"] Aug 04 19:49:05.796786 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.796766 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8129f458-75fb475468-ptz27"] Aug 04 19:49:05.811768 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:05.811745 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:07.603994 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:07.603961 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e935cf-361a-4e4c-b680-bc467c960f65" path="/var/lib/kubelet/pods/07e935cf-361a-4e4c-b680-bc467c960f65/volumes" Aug 04 19:49:07.604479 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:07.604411 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b67391-5b70-4ff1-947c-89d6cacc78dd" path="/var/lib/kubelet/pods/08b67391-5b70-4ff1-947c-89d6cacc78dd/volumes" Aug 04 19:49:11.991705 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:11.991671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:12.092187 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:12.092145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:12.273286 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:12.273188 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:12.514317 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:12.514282 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:12.670221 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:12.670118 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:12.782146 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:12.782116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:12.888861 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:12.888832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:18.644324 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:18.644275 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:18.707021 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:18.706990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:18.770414 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:18.770382 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:18.868997 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:18.868963 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:19.002332 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:19.002294 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:24.858743 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:24.858691 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:24.958644 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:24.958596 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:49:33.243027 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:49:33.242992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:50:55.637359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:50:55.637321 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:50:55.642303 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:50:55.642281 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:51:27.744426 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:27.744391 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:51:28.017389 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:28.017299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:51:28.128517 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:28.128479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:51:31.177292 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:31.177255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:51:31.274565 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:31.274513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:51:32.381859 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:32.381824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:51:32.507603 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:32.507569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:51:32.938406 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:32.938377 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:51:33.035607 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.035576 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:51:33.092763 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.092735 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq"] Aug 04 19:51:33.093099 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.093088 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b67391-5b70-4ff1-947c-89d6cacc78dd" containerName="payload-processing" Aug 04 19:51:33.093144 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.093101 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b67391-5b70-4ff1-947c-89d6cacc78dd" containerName="payload-processing" Aug 04 19:51:33.093144 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.093120 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e935cf-361a-4e4c-b680-bc467c960f65" containerName="payload-pre-processing" Aug 04 19:51:33.093144 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.093126 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e935cf-361a-4e4c-b680-bc467c960f65" containerName="payload-pre-processing" Aug 04 19:51:33.093239 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.093188 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b67391-5b70-4ff1-947c-89d6cacc78dd" containerName="payload-processing" Aug 04 19:51:33.093239 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.093198 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e935cf-361a-4e4c-b680-bc467c960f65" containerName="payload-pre-processing" Aug 04 19:51:33.096067 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.096051 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:51:33.099925 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.099906 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fb61cc13-dockercfg-w8xjc\"" Aug 04 19:51:33.100046 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.099954 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fb61cc13\"" Aug 04 19:51:33.110796 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.110772 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq"] Aug 04 19:51:33.125818 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.125789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87392b5f-2450-4160-94a9-80c2b6179a57-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq\" (UID: \"87392b5f-2450-4160-94a9-80c2b6179a57\") " pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:51:33.125961 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.125832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqhf\" (UniqueName: \"kubernetes.io/projected/87392b5f-2450-4160-94a9-80c2b6179a57-kube-api-access-pnqhf\") pod \"payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq\" (UID: \"87392b5f-2450-4160-94a9-80c2b6179a57\") " pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:51:33.154215 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.154184 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d"] Aug 04 19:51:33.157379 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.157364 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:51:33.171523 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.171447 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d"] Aug 04 19:51:33.226668 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.226637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87392b5f-2450-4160-94a9-80c2b6179a57-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq\" (UID: \"87392b5f-2450-4160-94a9-80c2b6179a57\") " pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:51:33.226881 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.226693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqhf\" (UniqueName: \"kubernetes.io/projected/87392b5f-2450-4160-94a9-80c2b6179a57-kube-api-access-pnqhf\") pod \"payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq\" (UID: \"87392b5f-2450-4160-94a9-80c2b6179a57\") " pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:51:33.226881 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.226769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ztt\" (UniqueName: \"kubernetes.io/projected/91e43584-c422-4b27-aa8b-49eb38a7e9ce-kube-api-access-26ztt\") pod \"payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d\" (UID: \"91e43584-c422-4b27-aa8b-49eb38a7e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:51:33.226881 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.226807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e43584-c422-4b27-aa8b-49eb38a7e9ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d\" (UID: \"91e43584-c422-4b27-aa8b-49eb38a7e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:51:33.227321 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.227302 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87392b5f-2450-4160-94a9-80c2b6179a57-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq\" (UID: \"87392b5f-2450-4160-94a9-80c2b6179a57\") " pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:51:33.236378 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.236355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqhf\" (UniqueName: \"kubernetes.io/projected/87392b5f-2450-4160-94a9-80c2b6179a57-kube-api-access-pnqhf\") pod \"payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq\" (UID: \"87392b5f-2450-4160-94a9-80c2b6179a57\") " pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:51:33.327517 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.327438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ztt\" (UniqueName: \"kubernetes.io/projected/91e43584-c422-4b27-aa8b-49eb38a7e9ce-kube-api-access-26ztt\") pod \"payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d\" (UID: \"91e43584-c422-4b27-aa8b-49eb38a7e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:51:33.327718 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.327541 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e43584-c422-4b27-aa8b-49eb38a7e9ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d\" (UID: \"91e43584-c422-4b27-aa8b-49eb38a7e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:51:33.328214 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.328193 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e43584-c422-4b27-aa8b-49eb38a7e9ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d\" (UID: \"91e43584-c422-4b27-aa8b-49eb38a7e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:51:33.338669 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.338640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ztt\" (UniqueName: \"kubernetes.io/projected/91e43584-c422-4b27-aa8b-49eb38a7e9ce-kube-api-access-26ztt\") pod \"payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d\" (UID: \"91e43584-c422-4b27-aa8b-49eb38a7e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:51:33.406639 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.406602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:51:33.467937 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.467898 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:51:33.556086 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.556057 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq"] Aug 04 19:51:33.558983 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:51:33.558950 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87392b5f_2450_4160_94a9_80c2b6179a57.slice/crio-8333e93a2fa99a46e8b57121616b994e2a74fbdfcaa75761181e85de2232b64b WatchSource:0}: Error finding container 8333e93a2fa99a46e8b57121616b994e2a74fbdfcaa75761181e85de2232b64b: Status 404 returned error can't find the container with id 8333e93a2fa99a46e8b57121616b994e2a74fbdfcaa75761181e85de2232b64b Aug 04 19:51:33.629292 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:33.629234 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d"] Aug 04 19:51:33.632795 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:51:33.632771 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e43584_c422_4b27_aa8b_49eb38a7e9ce.slice/crio-8d9f23432dc16e6e9caeca0dc63c3623e2b77fe95c7ae3c53afccc31e3f97a93 WatchSource:0}: Error finding container 8d9f23432dc16e6e9caeca0dc63c3623e2b77fe95c7ae3c53afccc31e3f97a93: Status 404 returned error can't find the container with id 8d9f23432dc16e6e9caeca0dc63c3623e2b77fe95c7ae3c53afccc31e3f97a93 Aug 04 19:51:34.452922 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:34.452877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" event={"ID":"91e43584-c422-4b27-aa8b-49eb38a7e9ce","Type":"ContainerStarted","Data":"a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468"} Aug 04 19:51:34.453378 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:34.453080 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:51:34.453378 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:34.453128 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" event={"ID":"91e43584-c422-4b27-aa8b-49eb38a7e9ce","Type":"ContainerStarted","Data":"8d9f23432dc16e6e9caeca0dc63c3623e2b77fe95c7ae3c53afccc31e3f97a93"} Aug 04 19:51:34.454557 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:34.454535 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" event={"ID":"87392b5f-2450-4160-94a9-80c2b6179a57","Type":"ContainerStarted","Data":"2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6"} Aug 04 19:51:34.454635 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:34.454577 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" event={"ID":"87392b5f-2450-4160-94a9-80c2b6179a57","Type":"ContainerStarted","Data":"8333e93a2fa99a46e8b57121616b994e2a74fbdfcaa75761181e85de2232b64b"} Aug 04 19:51:34.454635 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:34.454625 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:51:34.473055 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:34.473006 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" podStartSLOduration=1.472992071 podStartE2EDuration="1.472992071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:51:34.472108327 +0000 UTC m=+2139.377708869" watchObservedRunningTime="2026-08-04 19:51:34.472992071 +0000 UTC m=+2139.378592610" Aug 04 19:51:34.515607 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:34.515553 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" podStartSLOduration=1.515536003 podStartE2EDuration="1.515536003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:51:34.515142801 +0000 UTC m=+2139.420743341" watchObservedRunningTime="2026-08-04 19:51:34.515536003 +0000 UTC m=+2139.421136544" Aug 04 19:51:45.461218 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:45.461186 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:51:45.461719 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:45.461271 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:51:56.201172 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:51:56.201095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:52:12.013248 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:12.013221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:52:33.951152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:33.951116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:52:54.507688 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:54.507656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:52:54.603267 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:54.603224 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:52:56.446732 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.446681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:52:56.470333 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.470302 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq"] Aug 04 19:52:56.470633 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.470574 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" podUID="87392b5f-2450-4160-94a9-80c2b6179a57" containerName="payload-processing" containerID="cri-o://2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6" gracePeriod=30 Aug 04 19:52:56.474699 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.474669 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d"] Aug 04 19:52:56.474980 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.474906 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" podUID="91e43584-c422-4b27-aa8b-49eb38a7e9ce" containerName="payload-pre-processing" containerID="cri-o://a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468" gracePeriod=30 Aug 04 19:52:56.509739 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.509708 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:52:56.563250 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.563212 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:52:56.622787 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.622756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:52:56.744222 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.744202 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:52:56.747259 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.747238 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:52:56.850272 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.850233 2580 generic.go:356] "Generic (PLEG): container finished" podID="91e43584-c422-4b27-aa8b-49eb38a7e9ce" containerID="a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468" exitCode=0 Aug 04 19:52:56.850506 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.850282 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" Aug 04 19:52:56.850506 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.850336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" event={"ID":"91e43584-c422-4b27-aa8b-49eb38a7e9ce","Type":"ContainerDied","Data":"a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468"} Aug 04 19:52:56.850506 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.850395 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d" event={"ID":"91e43584-c422-4b27-aa8b-49eb38a7e9ce","Type":"ContainerDied","Data":"8d9f23432dc16e6e9caeca0dc63c3623e2b77fe95c7ae3c53afccc31e3f97a93"} Aug 04 19:52:56.850506 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.850418 2580 scope.go:117] "RemoveContainer" containerID="a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468" Aug 04 19:52:56.851710 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.851681 2580 generic.go:356] "Generic (PLEG): container finished" podID="87392b5f-2450-4160-94a9-80c2b6179a57" containerID="2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6" exitCode=0 Aug 04 19:52:56.851789 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.851736 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" Aug 04 19:52:56.851789 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.851774 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" event={"ID":"87392b5f-2450-4160-94a9-80c2b6179a57","Type":"ContainerDied","Data":"2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6"} Aug 04 19:52:56.851886 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.851823 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq" event={"ID":"87392b5f-2450-4160-94a9-80c2b6179a57","Type":"ContainerDied","Data":"8333e93a2fa99a46e8b57121616b994e2a74fbdfcaa75761181e85de2232b64b"} Aug 04 19:52:56.858953 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.858937 2580 scope.go:117] "RemoveContainer" containerID="a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468" Aug 04 19:52:56.859168 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:52:56.859151 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468\": container with ID starting with a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468 not found: ID does not exist" containerID="a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468" Aug 04 19:52:56.859219 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.859176 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468"} err="failed to get container status \"a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468\": rpc error: code = NotFound desc = could not find container \"a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468\": container with ID starting with a17aa1c513f800d9a2fc19bfffcdab0f95c2f892c860f07bebd5a13beb6e3468 not found: ID does not exist" Aug 04 19:52:56.859219 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.859198 2580 scope.go:117] "RemoveContainer" containerID="2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6" Aug 04 19:52:56.866990 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.866972 2580 scope.go:117] "RemoveContainer" containerID="2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6" Aug 04 19:52:56.867238 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:52:56.867221 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6\": container with ID starting with 2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6 not found: ID does not exist" containerID="2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6" Aug 04 19:52:56.867275 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.867245 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6"} err="failed to get container status \"2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6\": rpc error: code = NotFound desc = could not find container \"2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6\": container with ID starting with 2ab49d70879f7708e1464ad0e68998c1305d836caee4e74418ed69efcbf08bb6 not found: ID does not exist" Aug 04 19:52:56.876588 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.876570 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnqhf\" (UniqueName: \"kubernetes.io/projected/87392b5f-2450-4160-94a9-80c2b6179a57-kube-api-access-pnqhf\") pod \"87392b5f-2450-4160-94a9-80c2b6179a57\" (UID: \"87392b5f-2450-4160-94a9-80c2b6179a57\") " Aug 04 19:52:56.876640 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.876626 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87392b5f-2450-4160-94a9-80c2b6179a57-plugins-config-volume\") pod \"87392b5f-2450-4160-94a9-80c2b6179a57\" (UID: \"87392b5f-2450-4160-94a9-80c2b6179a57\") " Aug 04 19:52:56.876683 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.876659 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e43584-c422-4b27-aa8b-49eb38a7e9ce-plugins-config-volume\") pod \"91e43584-c422-4b27-aa8b-49eb38a7e9ce\" (UID: \"91e43584-c422-4b27-aa8b-49eb38a7e9ce\") " Aug 04 19:52:56.876717 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.876696 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26ztt\" (UniqueName: \"kubernetes.io/projected/91e43584-c422-4b27-aa8b-49eb38a7e9ce-kube-api-access-26ztt\") pod \"91e43584-c422-4b27-aa8b-49eb38a7e9ce\" (UID: \"91e43584-c422-4b27-aa8b-49eb38a7e9ce\") " Aug 04 19:52:56.877031 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.876994 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87392b5f-2450-4160-94a9-80c2b6179a57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87392b5f-2450-4160-94a9-80c2b6179a57" (UID: "87392b5f-2450-4160-94a9-80c2b6179a57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:52:56.877031 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.877010 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e43584-c422-4b27-aa8b-49eb38a7e9ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91e43584-c422-4b27-aa8b-49eb38a7e9ce" (UID: "91e43584-c422-4b27-aa8b-49eb38a7e9ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:52:56.878862 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.878834 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87392b5f-2450-4160-94a9-80c2b6179a57-kube-api-access-pnqhf" (OuterVolumeSpecName: "kube-api-access-pnqhf") pod "87392b5f-2450-4160-94a9-80c2b6179a57" (UID: "87392b5f-2450-4160-94a9-80c2b6179a57"). InnerVolumeSpecName "kube-api-access-pnqhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:52:56.878957 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.878937 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e43584-c422-4b27-aa8b-49eb38a7e9ce-kube-api-access-26ztt" (OuterVolumeSpecName: "kube-api-access-26ztt") pod "91e43584-c422-4b27-aa8b-49eb38a7e9ce" (UID: "91e43584-c422-4b27-aa8b-49eb38a7e9ce"). InnerVolumeSpecName "kube-api-access-26ztt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:52:56.977770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.977679 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqhf\" (UniqueName: \"kubernetes.io/projected/87392b5f-2450-4160-94a9-80c2b6179a57-kube-api-access-pnqhf\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:52:56.977770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.977711 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87392b5f-2450-4160-94a9-80c2b6179a57-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:52:56.977770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.977724 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e43584-c422-4b27-aa8b-49eb38a7e9ce-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:52:56.977770 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:56.977739 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26ztt\" (UniqueName: \"kubernetes.io/projected/91e43584-c422-4b27-aa8b-49eb38a7e9ce-kube-api-access-26ztt\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:52:57.173648 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:57.173617 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq"] Aug 04 19:52:57.177125 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:57.177100 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb61cc13-5b858b74fd-qkhtq"] Aug 04 19:52:57.186847 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:57.186827 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d"] Aug 04 19:52:57.190636 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:57.190617 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb61cc13-888b768b4-j4n7d"] Aug 04 19:52:57.603977 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:57.603944 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87392b5f-2450-4160-94a9-80c2b6179a57" path="/var/lib/kubelet/pods/87392b5f-2450-4160-94a9-80c2b6179a57/volumes" Aug 04 19:52:57.604339 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:57.604312 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e43584-c422-4b27-aa8b-49eb38a7e9ce" path="/var/lib/kubelet/pods/91e43584-c422-4b27-aa8b-49eb38a7e9ce/volumes" Aug 04 19:52:59.724859 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:52:59.724824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:53:04.802065 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:53:04.802027 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:53:41.574143 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:53:41.574111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:53:41.901393 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:53:41.901308 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:53:47.484447 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:53:47.484405 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:53:47.588860 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:53:47.588826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:53:47.770612 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:53:47.770531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:53:47.988294 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:53:47.988263 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:53:59.266748 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:53:59.266714 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:53:59.369104 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:53:59.369062 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:53:59.468258 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:53:59.468224 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:00.099991 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:00.099959 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:00.201231 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:00.201194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:00.300276 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:00.300245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:01.562577 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:01.562545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:08.101971 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:08.101932 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:08.202816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:08.202784 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:08.825942 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:08.825912 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:14.395574 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:14.395542 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:14.727804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:14.727775 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:14.830168 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:14.830135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:14.921187 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:14.921151 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:16.858760 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:16.858723 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:16.960458 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:16.960423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:25.429780 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:25.429748 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:25.662910 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:25.662875 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:26.809623 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:26.809591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:38.238898 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:38.238863 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:46.649956 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:46.649882 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:52.577947 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:52.577897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:52.685939 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:52.685905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:52.954391 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:52.954364 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:53.269693 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:53.269614 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:53.356642 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:53.356603 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:56.240402 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:56.240365 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:56.335194 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:56.335164 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:57.355122 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.355087 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:57.850180 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.850149 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:57.950533 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.950459 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:57.996213 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.996173 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8"] Aug 04 19:54:57.996660 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.996642 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87392b5f-2450-4160-94a9-80c2b6179a57" containerName="payload-processing" Aug 04 19:54:57.996760 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.996663 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87392b5f-2450-4160-94a9-80c2b6179a57" containerName="payload-processing" Aug 04 19:54:57.996760 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.996704 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e43584-c422-4b27-aa8b-49eb38a7e9ce" containerName="payload-pre-processing" Aug 04 19:54:57.996760 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.996714 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e43584-c422-4b27-aa8b-49eb38a7e9ce" containerName="payload-pre-processing" Aug 04 19:54:57.996913 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.996799 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87392b5f-2450-4160-94a9-80c2b6179a57" containerName="payload-processing" Aug 04 19:54:57.996913 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.996817 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e43584-c422-4b27-aa8b-49eb38a7e9ce" containerName="payload-pre-processing" Aug 04 19:54:57.999774 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:57.999755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:54:58.003806 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.003787 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-48bed7\"" Aug 04 19:54:58.003907 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.003839 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-48bed7-dockercfg-k8jz4\"" Aug 04 19:54:58.008569 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.008549 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8"] Aug 04 19:54:58.027228 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.027194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:54:58.037869 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.037846 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f"] Aug 04 19:54:58.041239 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.041224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:54:58.049362 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.049342 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f"] Aug 04 19:54:58.061980 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.061950 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6618782d-c422-458a-878a-8a3a3098d5a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8\" (UID: \"6618782d-c422-458a-878a-8a3a3098d5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:54:58.062082 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.062061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksm82\" (UniqueName: \"kubernetes.io/projected/6618782d-c422-458a-878a-8a3a3098d5a1-kube-api-access-ksm82\") pod \"payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8\" (UID: \"6618782d-c422-458a-878a-8a3a3098d5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:54:58.163317 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.163224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4dz\" (UniqueName: \"kubernetes.io/projected/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-kube-api-access-fx4dz\") pod \"payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f\" (UID: \"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:54:58.163317 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.163278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6618782d-c422-458a-878a-8a3a3098d5a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8\" (UID: \"6618782d-c422-458a-878a-8a3a3098d5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:54:58.163600 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.163371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksm82\" (UniqueName: \"kubernetes.io/projected/6618782d-c422-458a-878a-8a3a3098d5a1-kube-api-access-ksm82\") pod \"payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8\" (UID: \"6618782d-c422-458a-878a-8a3a3098d5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:54:58.163600 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.163506 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f\" (UID: \"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:54:58.164000 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.163982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6618782d-c422-458a-878a-8a3a3098d5a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8\" (UID: \"6618782d-c422-458a-878a-8a3a3098d5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:54:58.171636 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.171612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksm82\" (UniqueName: \"kubernetes.io/projected/6618782d-c422-458a-878a-8a3a3098d5a1-kube-api-access-ksm82\") pod \"payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8\" (UID: \"6618782d-c422-458a-878a-8a3a3098d5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:54:58.264966 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.264924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f\" (UID: \"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:54:58.264966 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.264966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4dz\" (UniqueName: \"kubernetes.io/projected/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-kube-api-access-fx4dz\") pod \"payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f\" (UID: \"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:54:58.265648 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.265628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f\" (UID: \"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:54:58.274950 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.274923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4dz\" (UniqueName: \"kubernetes.io/projected/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-kube-api-access-fx4dz\") pod \"payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f\" (UID: \"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:54:58.310872 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.310844 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:54:58.352044 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.351994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:54:58.471642 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.471604 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8"] Aug 04 19:54:58.474975 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:54:58.474939 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6618782d_c422_458a_878a_8a3a3098d5a1.slice/crio-abce91ed64e9e870c1c8a37ef67d742c37147766d44e998dee618823bd79b439 WatchSource:0}: Error finding container abce91ed64e9e870c1c8a37ef67d742c37147766d44e998dee618823bd79b439: Status 404 returned error can't find the container with id abce91ed64e9e870c1c8a37ef67d742c37147766d44e998dee618823bd79b439 Aug 04 19:54:58.542288 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:58.542261 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f"] Aug 04 19:54:58.545176 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:54:58.545148 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1117a2d1_d3db_4750_a0f5_0b3a1164eb6d.slice/crio-0531b8410da7b7ea12348d546b5179bb2ead95c9d42215fd4210655015bcf1ba WatchSource:0}: Error finding container 0531b8410da7b7ea12348d546b5179bb2ead95c9d42215fd4210655015bcf1ba: Status 404 returned error can't find the container with id 0531b8410da7b7ea12348d546b5179bb2ead95c9d42215fd4210655015bcf1ba Aug 04 19:54:59.451030 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:59.450996 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" event={"ID":"6618782d-c422-458a-878a-8a3a3098d5a1","Type":"ContainerStarted","Data":"78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f"} Aug 04 19:54:59.451212 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:59.451049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" event={"ID":"6618782d-c422-458a-878a-8a3a3098d5a1","Type":"ContainerStarted","Data":"abce91ed64e9e870c1c8a37ef67d742c37147766d44e998dee618823bd79b439"} Aug 04 19:54:59.451212 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:59.451114 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:54:59.452679 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:59.452654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" event={"ID":"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d","Type":"ContainerStarted","Data":"4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289"} Aug 04 19:54:59.452816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:59.452694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" event={"ID":"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d","Type":"ContainerStarted","Data":"0531b8410da7b7ea12348d546b5179bb2ead95c9d42215fd4210655015bcf1ba"} Aug 04 19:54:59.452816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:59.452726 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:54:59.465942 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:59.465900 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" podStartSLOduration=2.46588066 podStartE2EDuration="2.46588066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:54:59.463616976 +0000 UTC m=+2344.369217522" watchObservedRunningTime="2026-08-04 19:54:59.46588066 +0000 UTC m=+2344.371481198" Aug 04 19:54:59.478044 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:54:59.478003 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" podStartSLOduration=1.477989203 podStartE2EDuration="1.477989203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:54:59.476652038 +0000 UTC m=+2344.382252578" watchObservedRunningTime="2026-08-04 19:54:59.477989203 +0000 UTC m=+2344.383589741" Aug 04 19:55:10.461137 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:10.461105 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:55:10.461697 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:10.461182 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:55:21.238083 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:21.238043 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:21.594846 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:21.594764 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:21.902291 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:21.902199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:22.116237 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:22.116206 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:24.449960 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:24.449931 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:24.549622 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:24.549595 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:25.653624 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:25.653593 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:25.959398 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:25.959366 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:26.212840 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.212740 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:26.311711 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.311677 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:26.320629 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.318012 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2"] Aug 04 19:55:26.323386 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.323363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:55:26.326364 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.326045 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ddb738\"" Aug 04 19:55:26.326364 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.326225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ddb738-dockercfg-6grzj\"" Aug 04 19:55:26.330311 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.330282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2"] Aug 04 19:55:26.342215 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.342193 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg"] Aug 04 19:55:26.345733 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.345714 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:55:26.357436 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.357407 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg"] Aug 04 19:55:26.411595 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.411566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pvvd\" (UniqueName: \"kubernetes.io/projected/88d8d170-3e2f-4662-b5c4-0416308bbe53-kube-api-access-5pvvd\") pod \"payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2\" (UID: \"88d8d170-3e2f-4662-b5c4-0416308bbe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:55:26.411769 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.411648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d8d170-3e2f-4662-b5c4-0416308bbe53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2\" (UID: \"88d8d170-3e2f-4662-b5c4-0416308bbe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:55:26.411769 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.411674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg\" (UID: \"1a8a6635-99b0-4c2e-84ac-c860c417ce0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:55:26.411769 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.411690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s27pv\" (UniqueName: \"kubernetes.io/projected/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-kube-api-access-s27pv\") pod \"payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg\" (UID: \"1a8a6635-99b0-4c2e-84ac-c860c417ce0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:55:26.512927 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.512827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pvvd\" (UniqueName: \"kubernetes.io/projected/88d8d170-3e2f-4662-b5c4-0416308bbe53-kube-api-access-5pvvd\") pod \"payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2\" (UID: \"88d8d170-3e2f-4662-b5c4-0416308bbe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:55:26.512927 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.512918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d8d170-3e2f-4662-b5c4-0416308bbe53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2\" (UID: \"88d8d170-3e2f-4662-b5c4-0416308bbe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:55:26.513196 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.512944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg\" (UID: \"1a8a6635-99b0-4c2e-84ac-c860c417ce0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:55:26.513196 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.512961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s27pv\" (UniqueName: \"kubernetes.io/projected/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-kube-api-access-s27pv\") pod \"payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg\" (UID: \"1a8a6635-99b0-4c2e-84ac-c860c417ce0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:55:26.513784 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.513759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg\" (UID: \"1a8a6635-99b0-4c2e-84ac-c860c417ce0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:55:26.513873 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.513814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d8d170-3e2f-4662-b5c4-0416308bbe53-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2\" (UID: \"88d8d170-3e2f-4662-b5c4-0416308bbe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:55:26.520867 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.520839 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pvvd\" (UniqueName: \"kubernetes.io/projected/88d8d170-3e2f-4662-b5c4-0416308bbe53-kube-api-access-5pvvd\") pod \"payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2\" (UID: \"88d8d170-3e2f-4662-b5c4-0416308bbe53\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:55:26.520867 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.520857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s27pv\" (UniqueName: \"kubernetes.io/projected/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-kube-api-access-s27pv\") pod \"payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg\" (UID: \"1a8a6635-99b0-4c2e-84ac-c860c417ce0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:55:26.636847 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.636808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:55:26.656149 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.656116 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:55:26.790239 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:26.790206 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2"] Aug 04 19:55:26.791706 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:55:26.791667 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d8d170_3e2f_4662_b5c4_0416308bbe53.slice/crio-f15d03d46aa095caa267b4b76e5ac85fadeec6e5a46314c06e45eb9e9804dad5 WatchSource:0}: Error finding container f15d03d46aa095caa267b4b76e5ac85fadeec6e5a46314c06e45eb9e9804dad5: Status 404 returned error can't find the container with id f15d03d46aa095caa267b4b76e5ac85fadeec6e5a46314c06e45eb9e9804dad5 Aug 04 19:55:27.022948 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:27.022918 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg"] Aug 04 19:55:27.025706 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:55:27.025684 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8a6635_99b0_4c2e_84ac_c860c417ce0d.slice/crio-ef9d1b95a6efef998daf92c263552eea3434ea3fce20172967a6cc508a152c51 WatchSource:0}: Error finding container ef9d1b95a6efef998daf92c263552eea3434ea3fce20172967a6cc508a152c51: Status 404 returned error can't find the container with id ef9d1b95a6efef998daf92c263552eea3434ea3fce20172967a6cc508a152c51 Aug 04 19:55:27.592566 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:27.592504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" event={"ID":"88d8d170-3e2f-4662-b5c4-0416308bbe53","Type":"ContainerStarted","Data":"59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3"} Aug 04 19:55:27.592774 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:27.592575 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" event={"ID":"88d8d170-3e2f-4662-b5c4-0416308bbe53","Type":"ContainerStarted","Data":"f15d03d46aa095caa267b4b76e5ac85fadeec6e5a46314c06e45eb9e9804dad5"} Aug 04 19:55:27.592774 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:27.592599 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:55:27.594219 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:27.594194 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" event={"ID":"1a8a6635-99b0-4c2e-84ac-c860c417ce0d","Type":"ContainerStarted","Data":"bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5"} Aug 04 19:55:27.594346 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:27.594241 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" event={"ID":"1a8a6635-99b0-4c2e-84ac-c860c417ce0d","Type":"ContainerStarted","Data":"ef9d1b95a6efef998daf92c263552eea3434ea3fce20172967a6cc508a152c51"} Aug 04 19:55:27.594346 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:27.594270 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:55:27.609684 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:27.609645 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" podStartSLOduration=1.60962947 podStartE2EDuration="1.60962947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:55:27.608194825 +0000 UTC m=+2372.513795365" watchObservedRunningTime="2026-08-04 19:55:27.60962947 +0000 UTC m=+2372.515230009" Aug 04 19:55:27.626098 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:27.626059 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" podStartSLOduration=1.626036372 podStartE2EDuration="1.626036372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:55:27.624364788 +0000 UTC m=+2372.529965330" watchObservedRunningTime="2026-08-04 19:55:27.626036372 +0000 UTC m=+2372.531636973" Aug 04 19:55:38.600828 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:38.600793 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:55:38.601187 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:38.600965 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:55:49.355402 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:49.355368 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:55:55.666347 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:55.666316 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:55:55.673272 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:55:55.673249 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 19:56:07.605253 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.605223 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46"] Aug 04 19:56:07.608658 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.608641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.611931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.611910 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-48bed7\"/\"test-model-48bed7-kserve-self-signed-certs\"" Aug 04 19:56:07.612834 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.612814 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-48bed7\"/\"kube-root-ca.crt\"" Aug 04 19:56:07.612971 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.612894 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-48bed7\"/\"default-dockercfg-tkt5p\"" Aug 04 19:56:07.619820 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.619802 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-48bed7\"/\"openshift-service-ca.crt\"" Aug 04 19:56:07.623944 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.623925 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46"] Aug 04 19:56:07.658987 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.658956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:56:07.690166 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.690137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-dshm\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.690350 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.690201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-model-cache\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.690350 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.690245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tls-certs\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.690350 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.690298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-home\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.690350 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.690321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62fn\" (UniqueName: \"kubernetes.io/projected/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-kube-api-access-h62fn\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.690556 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.690375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tmp-dir\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.790808 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.790780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-model-cache\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.790808 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.790811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tls-certs\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.791063 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.790852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-home\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.791063 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.790969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62fn\" (UniqueName: \"kubernetes.io/projected/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-kube-api-access-h62fn\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.791063 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.791020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tmp-dir\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.791227 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.791080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-dshm\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.791227 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.791163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-model-cache\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.791227 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.791201 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-home\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.791351 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.791329 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tmp-dir\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.793290 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.793268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-dshm\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.793566 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.793549 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tls-certs\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.802314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.802294 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62fn\" (UniqueName: \"kubernetes.io/projected/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-kube-api-access-h62fn\") pod \"test-model-48bed7-kserve-549c955785-2vj46\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:07.918798 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:07.918710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:08.056801 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:08.056720 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46"] Aug 04 19:56:08.059149 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:56:08.059117 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d9a3ca0_a73b_4a87_ae38_02ffae698ff3.slice/crio-3e61e04d905073f7528202c2c0063a253eaba0235223d5049092a6cac979e199 WatchSource:0}: Error finding container 3e61e04d905073f7528202c2c0063a253eaba0235223d5049092a6cac979e199: Status 404 returned error can't find the container with id 3e61e04d905073f7528202c2c0063a253eaba0235223d5049092a6cac979e199 Aug 04 19:56:08.789405 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:08.789367 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" event={"ID":"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3","Type":"ContainerStarted","Data":"23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a"} Aug 04 19:56:08.789405 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:08.789411 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" event={"ID":"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3","Type":"ContainerStarted","Data":"3e61e04d905073f7528202c2c0063a253eaba0235223d5049092a6cac979e199"} Aug 04 19:56:08.808364 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:08.808315 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" podStartSLOduration=1.808299025 podStartE2EDuration="1.808299025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:56:08.806214651 +0000 UTC m=+2413.711815190" watchObservedRunningTime="2026-08-04 19:56:08.808299025 +0000 UTC m=+2413.713899564" Aug 04 19:56:17.919763 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:17.919665 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:17.919763 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:17.919723 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:17.932389 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:17.932361 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:18.848965 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:18.848936 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:56:26.618480 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:56:26.618435 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:09.174925 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:09.174890 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46"] Aug 04 19:57:09.194115 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:09.194079 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:10.061568 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:10.061524 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:20.629111 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:20.629072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:20.652711 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:20.652675 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f"] Aug 04 19:57:20.652886 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:20.652850 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" podUID="1117a2d1-d3db-4750-a0f5-0b3a1164eb6d" containerName="payload-processing" containerID="cri-o://4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289" gracePeriod=30 Aug 04 19:57:20.659405 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:20.659378 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8"] Aug 04 19:57:20.659632 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:20.659609 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" podUID="6618782d-c422-458a-878a-8a3a3098d5a1" containerName="payload-pre-processing" containerID="cri-o://78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f" gracePeriod=30 Aug 04 19:57:20.692721 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:20.692695 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:20.753498 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:20.753454 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:20.815170 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:20.815136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:20.933574 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:20.933553 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:57:20.936366 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:20.936349 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:57:21.054037 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.054011 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksm82\" (UniqueName: \"kubernetes.io/projected/6618782d-c422-458a-878a-8a3a3098d5a1-kube-api-access-ksm82\") pod \"6618782d-c422-458a-878a-8a3a3098d5a1\" (UID: \"6618782d-c422-458a-878a-8a3a3098d5a1\") " Aug 04 19:57:21.054208 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.054107 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-plugins-config-volume\") pod \"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d\" (UID: \"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d\") " Aug 04 19:57:21.054208 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.054140 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx4dz\" (UniqueName: \"kubernetes.io/projected/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-kube-api-access-fx4dz\") pod \"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d\" (UID: \"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d\") " Aug 04 19:57:21.054208 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.054206 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6618782d-c422-458a-878a-8a3a3098d5a1-plugins-config-volume\") pod \"6618782d-c422-458a-878a-8a3a3098d5a1\" (UID: \"6618782d-c422-458a-878a-8a3a3098d5a1\") " Aug 04 19:57:21.054541 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.054506 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1117a2d1-d3db-4750-a0f5-0b3a1164eb6d" (UID: "1117a2d1-d3db-4750-a0f5-0b3a1164eb6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:57:21.054664 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.054636 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6618782d-c422-458a-878a-8a3a3098d5a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6618782d-c422-458a-878a-8a3a3098d5a1" (UID: "6618782d-c422-458a-878a-8a3a3098d5a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:57:21.056305 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.056285 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6618782d-c422-458a-878a-8a3a3098d5a1-kube-api-access-ksm82" (OuterVolumeSpecName: "kube-api-access-ksm82") pod "6618782d-c422-458a-878a-8a3a3098d5a1" (UID: "6618782d-c422-458a-878a-8a3a3098d5a1"). InnerVolumeSpecName "kube-api-access-ksm82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:21.056421 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.056312 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-kube-api-access-fx4dz" (OuterVolumeSpecName: "kube-api-access-fx4dz") pod "1117a2d1-d3db-4750-a0f5-0b3a1164eb6d" (UID: "1117a2d1-d3db-4750-a0f5-0b3a1164eb6d"). InnerVolumeSpecName "kube-api-access-fx4dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:21.151523 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.151494 2580 generic.go:356] "Generic (PLEG): container finished" podID="6618782d-c422-458a-878a-8a3a3098d5a1" containerID="78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f" exitCode=0 Aug 04 19:57:21.151666 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.151574 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" event={"ID":"6618782d-c422-458a-878a-8a3a3098d5a1","Type":"ContainerDied","Data":"78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f"} Aug 04 19:57:21.151666 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.151598 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" Aug 04 19:57:21.151666 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.151635 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8" event={"ID":"6618782d-c422-458a-878a-8a3a3098d5a1","Type":"ContainerDied","Data":"abce91ed64e9e870c1c8a37ef67d742c37147766d44e998dee618823bd79b439"} Aug 04 19:57:21.151799 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.151670 2580 scope.go:117] "RemoveContainer" containerID="78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f" Aug 04 19:57:21.152876 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.152858 2580 generic.go:356] "Generic (PLEG): container finished" podID="1117a2d1-d3db-4750-a0f5-0b3a1164eb6d" containerID="4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289" exitCode=0 Aug 04 19:57:21.152942 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.152914 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" Aug 04 19:57:21.152942 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.152925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" event={"ID":"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d","Type":"ContainerDied","Data":"4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289"} Aug 04 19:57:21.153029 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.152986 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f" event={"ID":"1117a2d1-d3db-4750-a0f5-0b3a1164eb6d","Type":"ContainerDied","Data":"0531b8410da7b7ea12348d546b5179bb2ead95c9d42215fd4210655015bcf1ba"} Aug 04 19:57:21.154935 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.154883 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksm82\" (UniqueName: \"kubernetes.io/projected/6618782d-c422-458a-878a-8a3a3098d5a1-kube-api-access-ksm82\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:21.154935 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.154909 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:21.154935 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.154923 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx4dz\" (UniqueName: \"kubernetes.io/projected/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d-kube-api-access-fx4dz\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:21.155109 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.154937 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6618782d-c422-458a-878a-8a3a3098d5a1-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:21.161668 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.161647 2580 scope.go:117] "RemoveContainer" containerID="78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f" Aug 04 19:57:21.161963 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:57:21.161942 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f\": container with ID starting with 78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f not found: ID does not exist" containerID="78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f" Aug 04 19:57:21.162144 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.161969 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f"} err="failed to get container status \"78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f\": rpc error: code = NotFound desc = could not find container \"78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f\": container with ID starting with 78b5299a3a11bf50e7b5d7b56211cd65e3a060bd4ba83b019d9c16c058e4999f not found: ID does not exist" Aug 04 19:57:21.162144 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.161986 2580 scope.go:117] "RemoveContainer" containerID="4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289" Aug 04 19:57:21.169781 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.169766 2580 scope.go:117] "RemoveContainer" containerID="4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289" Aug 04 19:57:21.170057 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:57:21.170038 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289\": container with ID starting with 4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289 not found: ID does not exist" containerID="4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289" Aug 04 19:57:21.170118 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.170063 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289"} err="failed to get container status \"4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289\": rpc error: code = NotFound desc = could not find container \"4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289\": container with ID starting with 4d5720caebdc60684cdf36323fef99eea501e90204961c98f3dfb80010db5289 not found: ID does not exist" Aug 04 19:57:21.177210 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.177186 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f"] Aug 04 19:57:21.182507 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.182485 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48bed7-7c7b798db6-rf26f"] Aug 04 19:57:21.191434 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.191415 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8"] Aug 04 19:57:21.193419 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.193400 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48bed7-7b99598d-vfln8"] Aug 04 19:57:21.197521 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.197505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:21.603397 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.603354 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1117a2d1-d3db-4750-a0f5-0b3a1164eb6d" path="/var/lib/kubelet/pods/1117a2d1-d3db-4750-a0f5-0b3a1164eb6d/volumes" Aug 04 19:57:21.603816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:21.603796 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6618782d-c422-458a-878a-8a3a3098d5a1" path="/var/lib/kubelet/pods/6618782d-c422-458a-878a-8a3a3098d5a1/volumes" Aug 04 19:57:24.195121 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.195063 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" podUID="4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" containerName="main" containerID="cri-o://23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a" gracePeriod=45 Aug 04 19:57:24.448152 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.448101 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:57:24.585151 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585125 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-dshm\") pod \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " Aug 04 19:57:24.585318 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585166 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tls-certs\") pod \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " Aug 04 19:57:24.585318 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585198 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h62fn\" (UniqueName: \"kubernetes.io/projected/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-kube-api-access-h62fn\") pod \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " Aug 04 19:57:24.585318 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585227 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-model-cache\") pod \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " Aug 04 19:57:24.585318 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585287 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-home\") pod \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " Aug 04 19:57:24.585524 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585361 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tmp-dir\") pod \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\" (UID: \"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3\") " Aug 04 19:57:24.585598 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585531 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" (UID: "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:24.585662 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585605 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-home" (OuterVolumeSpecName: "home") pod "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" (UID: "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:24.585717 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585683 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" (UID: "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:24.585820 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585800 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tmp-dir\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:24.585905 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585826 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-model-cache\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:24.585905 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.585840 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-home\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:24.587445 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.587421 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" (UID: "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:57:24.587553 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.587490 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-dshm" (OuterVolumeSpecName: "dshm") pod "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" (UID: "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:24.587641 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.587619 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-kube-api-access-h62fn" (OuterVolumeSpecName: "kube-api-access-h62fn") pod "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" (UID: "4d9a3ca0-a73b-4a87-ae38-02ffae698ff3"). InnerVolumeSpecName "kube-api-access-h62fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:24.686715 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.686681 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-dshm\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:24.686715 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.686708 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-tls-certs\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:24.686715 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:24.686724 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h62fn\" (UniqueName: \"kubernetes.io/projected/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3-kube-api-access-h62fn\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:25.174773 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:25.174739 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" containerID="23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a" exitCode=0 Aug 04 19:57:25.174938 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:25.174779 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" Aug 04 19:57:25.174938 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:25.174804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" event={"ID":"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3","Type":"ContainerDied","Data":"23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a"} Aug 04 19:57:25.174938 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:25.174839 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46" event={"ID":"4d9a3ca0-a73b-4a87-ae38-02ffae698ff3","Type":"ContainerDied","Data":"3e61e04d905073f7528202c2c0063a253eaba0235223d5049092a6cac979e199"} Aug 04 19:57:25.174938 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:25.174856 2580 scope.go:117] "RemoveContainer" containerID="23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a" Aug 04 19:57:25.184079 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:25.184062 2580 scope.go:117] "RemoveContainer" containerID="23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a" Aug 04 19:57:25.184345 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:57:25.184326 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a\": container with ID starting with 23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a not found: ID does not exist" containerID="23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a" Aug 04 19:57:25.184443 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:25.184351 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a"} err="failed to get container status \"23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a\": rpc error: code = NotFound desc = could not find container \"23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a\": container with ID starting with 23f04b56cf236d608c60da912e9ba6b2d20dd2bbb6b234b4db9b1c29e8981e7a not found: ID does not exist" Aug 04 19:57:25.195229 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:25.195204 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46"] Aug 04 19:57:25.198684 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:25.198665 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-48bed7/test-model-48bed7-kserve-549c955785-2vj46"] Aug 04 19:57:25.604110 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:25.604075 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" path="/var/lib/kubelet/pods/4d9a3ca0-a73b-4a87-ae38-02ffae698ff3/volumes" Aug 04 19:57:27.055651 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:27.055616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:27.110625 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:27.110596 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:27.340800 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:27.340710 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:27.596093 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:27.596013 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:38.142083 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.142051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:38.170449 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.170423 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg"] Aug 04 19:57:38.170707 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.170673 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" podUID="1a8a6635-99b0-4c2e-84ac-c860c417ce0d" containerName="payload-processing" containerID="cri-o://bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5" gracePeriod=30 Aug 04 19:57:38.177632 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.177610 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2"] Aug 04 19:57:38.177851 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.177827 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" podUID="88d8d170-3e2f-4662-b5c4-0416308bbe53" containerName="payload-pre-processing" containerID="cri-o://59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3" gracePeriod=30 Aug 04 19:57:38.211316 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.211285 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:38.270185 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.270151 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:38.371727 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.371691 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:38.454389 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.454358 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:57:38.457266 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.457249 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:57:38.604608 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.604574 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s27pv\" (UniqueName: \"kubernetes.io/projected/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-kube-api-access-s27pv\") pod \"1a8a6635-99b0-4c2e-84ac-c860c417ce0d\" (UID: \"1a8a6635-99b0-4c2e-84ac-c860c417ce0d\") " Aug 04 19:57:38.604754 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.604648 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pvvd\" (UniqueName: \"kubernetes.io/projected/88d8d170-3e2f-4662-b5c4-0416308bbe53-kube-api-access-5pvvd\") pod \"88d8d170-3e2f-4662-b5c4-0416308bbe53\" (UID: \"88d8d170-3e2f-4662-b5c4-0416308bbe53\") " Aug 04 19:57:38.604754 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.604698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d8d170-3e2f-4662-b5c4-0416308bbe53-plugins-config-volume\") pod \"88d8d170-3e2f-4662-b5c4-0416308bbe53\" (UID: \"88d8d170-3e2f-4662-b5c4-0416308bbe53\") " Aug 04 19:57:38.604754 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.604725 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-plugins-config-volume\") pod \"1a8a6635-99b0-4c2e-84ac-c860c417ce0d\" (UID: \"1a8a6635-99b0-4c2e-84ac-c860c417ce0d\") " Aug 04 19:57:38.605071 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.605048 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d8d170-3e2f-4662-b5c4-0416308bbe53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88d8d170-3e2f-4662-b5c4-0416308bbe53" (UID: "88d8d170-3e2f-4662-b5c4-0416308bbe53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:57:38.605133 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.605094 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a8a6635-99b0-4c2e-84ac-c860c417ce0d" (UID: "1a8a6635-99b0-4c2e-84ac-c860c417ce0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:57:38.606900 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.606879 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d8d170-3e2f-4662-b5c4-0416308bbe53-kube-api-access-5pvvd" (OuterVolumeSpecName: "kube-api-access-5pvvd") pod "88d8d170-3e2f-4662-b5c4-0416308bbe53" (UID: "88d8d170-3e2f-4662-b5c4-0416308bbe53"). InnerVolumeSpecName "kube-api-access-5pvvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:38.606972 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.606952 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-kube-api-access-s27pv" (OuterVolumeSpecName: "kube-api-access-s27pv") pod "1a8a6635-99b0-4c2e-84ac-c860c417ce0d" (UID: "1a8a6635-99b0-4c2e-84ac-c860c417ce0d"). InnerVolumeSpecName "kube-api-access-s27pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:38.706421 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.706206 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pvvd\" (UniqueName: \"kubernetes.io/projected/88d8d170-3e2f-4662-b5c4-0416308bbe53-kube-api-access-5pvvd\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:38.706421 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.706237 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d8d170-3e2f-4662-b5c4-0416308bbe53-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:38.706421 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.706254 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:38.706421 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.706266 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s27pv\" (UniqueName: \"kubernetes.io/projected/1a8a6635-99b0-4c2e-84ac-c860c417ce0d-kube-api-access-s27pv\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:57:38.731036 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:38.731009 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:39.243941 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.243910 2580 generic.go:356] "Generic (PLEG): container finished" podID="88d8d170-3e2f-4662-b5c4-0416308bbe53" containerID="59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3" exitCode=0 Aug 04 19:57:39.244437 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.243953 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" Aug 04 19:57:39.244437 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.243996 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" event={"ID":"88d8d170-3e2f-4662-b5c4-0416308bbe53","Type":"ContainerDied","Data":"59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3"} Aug 04 19:57:39.244437 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.244060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2" event={"ID":"88d8d170-3e2f-4662-b5c4-0416308bbe53","Type":"ContainerDied","Data":"f15d03d46aa095caa267b4b76e5ac85fadeec6e5a46314c06e45eb9e9804dad5"} Aug 04 19:57:39.244437 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.244094 2580 scope.go:117] "RemoveContainer" containerID="59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3" Aug 04 19:57:39.245491 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.245291 2580 generic.go:356] "Generic (PLEG): container finished" podID="1a8a6635-99b0-4c2e-84ac-c860c417ce0d" containerID="bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5" exitCode=0 Aug 04 19:57:39.245491 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.245365 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" event={"ID":"1a8a6635-99b0-4c2e-84ac-c860c417ce0d","Type":"ContainerDied","Data":"bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5"} Aug 04 19:57:39.245491 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.245416 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" Aug 04 19:57:39.245638 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.245416 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg" event={"ID":"1a8a6635-99b0-4c2e-84ac-c860c417ce0d","Type":"ContainerDied","Data":"ef9d1b95a6efef998daf92c263552eea3434ea3fce20172967a6cc508a152c51"} Aug 04 19:57:39.253875 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.253860 2580 scope.go:117] "RemoveContainer" containerID="59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3" Aug 04 19:57:39.254113 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:57:39.254093 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3\": container with ID starting with 59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3 not found: ID does not exist" containerID="59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3" Aug 04 19:57:39.254198 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.254120 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3"} err="failed to get container status \"59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3\": rpc error: code = NotFound desc = could not find container \"59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3\": container with ID starting with 59689dc853afaaa52440545a99a32ba43c7d4c025b893730fcd10d3e75b6a5b3 not found: ID does not exist" Aug 04 19:57:39.254198 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.254137 2580 scope.go:117] "RemoveContainer" containerID="bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5" Aug 04 19:57:39.262627 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.262612 2580 scope.go:117] "RemoveContainer" containerID="bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5" Aug 04 19:57:39.262886 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:57:39.262870 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5\": container with ID starting with bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5 not found: ID does not exist" containerID="bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5" Aug 04 19:57:39.262942 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.262894 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5"} err="failed to get container status \"bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5\": rpc error: code = NotFound desc = could not find container \"bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5\": container with ID starting with bfcaaa634edaf72b77a2521028f5c1313d1946812642568020179f84eac9ddf5 not found: ID does not exist" Aug 04 19:57:39.270558 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.270534 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg"] Aug 04 19:57:39.274122 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.274102 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb738-5f794b8c68-2wrsg"] Aug 04 19:57:39.291328 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.291303 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2"] Aug 04 19:57:39.295627 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.295607 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb738-58bd465bdd-rvtx2"] Aug 04 19:57:39.603586 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.603517 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8a6635-99b0-4c2e-84ac-c860c417ce0d" path="/var/lib/kubelet/pods/1a8a6635-99b0-4c2e-84ac-c860c417ce0d/volumes" Aug 04 19:57:39.603930 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:39.603917 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d8d170-3e2f-4662-b5c4-0416308bbe53" path="/var/lib/kubelet/pods/88d8d170-3e2f-4662-b5c4-0416308bbe53/volumes" Aug 04 19:57:44.474909 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:44.474870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:44.572063 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:44.572034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:44.719264 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:44.719232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:44.957663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:44.957631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:45.304809 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:45.304727 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:45.620426 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:45.620352 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:45.803382 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:45.803349 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:48.076589 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:48.076511 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:48.174374 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:48.174341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:49.182973 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.182940 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:49.483840 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.483802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:49.812813 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.812724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:49.912457 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.912428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:57:49.917384 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917359 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j"] Aug 04 19:57:49.917767 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917754 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8a6635-99b0-4c2e-84ac-c860c417ce0d" containerName="payload-processing" Aug 04 19:57:49.917816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917769 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8a6635-99b0-4c2e-84ac-c860c417ce0d" containerName="payload-processing" Aug 04 19:57:49.917816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917786 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6618782d-c422-458a-878a-8a3a3098d5a1" containerName="payload-pre-processing" Aug 04 19:57:49.917816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917791 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6618782d-c422-458a-878a-8a3a3098d5a1" containerName="payload-pre-processing" Aug 04 19:57:49.917816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917797 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1117a2d1-d3db-4750-a0f5-0b3a1164eb6d" containerName="payload-processing" Aug 04 19:57:49.917816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917802 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1117a2d1-d3db-4750-a0f5-0b3a1164eb6d" containerName="payload-processing" Aug 04 19:57:49.917816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917810 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" containerName="main" Aug 04 19:57:49.917816 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917815 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" containerName="main" Aug 04 19:57:49.918041 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917824 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d8d170-3e2f-4662-b5c4-0416308bbe53" containerName="payload-pre-processing" Aug 04 19:57:49.918041 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917830 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d8d170-3e2f-4662-b5c4-0416308bbe53" containerName="payload-pre-processing" Aug 04 19:57:49.918041 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917891 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6618782d-c422-458a-878a-8a3a3098d5a1" containerName="payload-pre-processing" Aug 04 19:57:49.918041 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917910 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9a3ca0-a73b-4a87-ae38-02ffae698ff3" containerName="main" Aug 04 19:57:49.918041 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917916 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8a6635-99b0-4c2e-84ac-c860c417ce0d" containerName="payload-processing" Aug 04 19:57:49.918041 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917923 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d8d170-3e2f-4662-b5c4-0416308bbe53" containerName="payload-pre-processing" Aug 04 19:57:49.918041 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.917930 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1117a2d1-d3db-4750-a0f5-0b3a1164eb6d" containerName="payload-processing" Aug 04 19:57:49.922555 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.922537 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:57:49.925065 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.925044 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4feefd-dockercfg-ffh8n\"" Aug 04 19:57:49.925178 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.925092 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4feefd\"" Aug 04 19:57:49.930843 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.930811 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j"] Aug 04 19:57:49.946982 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.946952 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw"] Aug 04 19:57:49.955821 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.955803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:57:49.965805 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:49.965777 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw"] Aug 04 19:57:50.003769 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.003736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wstg\" (UniqueName: \"kubernetes.io/projected/dba231ea-db7c-49f6-b435-d87f4457156f-kube-api-access-6wstg\") pod \"payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j\" (UID: \"dba231ea-db7c-49f6-b435-d87f4457156f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:57:50.003930 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.003810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dba231ea-db7c-49f6-b435-d87f4457156f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j\" (UID: \"dba231ea-db7c-49f6-b435-d87f4457156f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:57:50.003980 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.003929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e810eb31-7159-4e96-8280-eab12bd4154a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw\" (UID: \"e810eb31-7159-4e96-8280-eab12bd4154a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:57:50.004028 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.004009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7922\" (UniqueName: \"kubernetes.io/projected/e810eb31-7159-4e96-8280-eab12bd4154a-kube-api-access-b7922\") pod \"payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw\" (UID: \"e810eb31-7159-4e96-8280-eab12bd4154a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:57:50.104886 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.104803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e810eb31-7159-4e96-8280-eab12bd4154a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw\" (UID: \"e810eb31-7159-4e96-8280-eab12bd4154a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:57:50.104886 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.104851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7922\" (UniqueName: \"kubernetes.io/projected/e810eb31-7159-4e96-8280-eab12bd4154a-kube-api-access-b7922\") pod \"payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw\" (UID: \"e810eb31-7159-4e96-8280-eab12bd4154a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:57:50.105064 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.104899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wstg\" (UniqueName: \"kubernetes.io/projected/dba231ea-db7c-49f6-b435-d87f4457156f-kube-api-access-6wstg\") pod \"payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j\" (UID: \"dba231ea-db7c-49f6-b435-d87f4457156f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:57:50.105064 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.104924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dba231ea-db7c-49f6-b435-d87f4457156f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j\" (UID: \"dba231ea-db7c-49f6-b435-d87f4457156f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:57:50.105594 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.105559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e810eb31-7159-4e96-8280-eab12bd4154a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw\" (UID: \"e810eb31-7159-4e96-8280-eab12bd4154a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:57:50.105832 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.105814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dba231ea-db7c-49f6-b435-d87f4457156f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j\" (UID: \"dba231ea-db7c-49f6-b435-d87f4457156f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:57:50.113667 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.113647 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wstg\" (UniqueName: \"kubernetes.io/projected/dba231ea-db7c-49f6-b435-d87f4457156f-kube-api-access-6wstg\") pod \"payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j\" (UID: \"dba231ea-db7c-49f6-b435-d87f4457156f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:57:50.113777 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.113704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7922\" (UniqueName: \"kubernetes.io/projected/e810eb31-7159-4e96-8280-eab12bd4154a-kube-api-access-b7922\") pod \"payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw\" (UID: \"e810eb31-7159-4e96-8280-eab12bd4154a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:57:50.233992 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.233951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:57:50.267424 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.267395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:57:50.389663 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.389630 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j"] Aug 04 19:57:50.391379 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:57:50.391349 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba231ea_db7c_49f6_b435_d87f4457156f.slice/crio-e555fa613c8e8bc8d033fbde7918e568fac0954756db0220d3de019456657b5f WatchSource:0}: Error finding container e555fa613c8e8bc8d033fbde7918e568fac0954756db0220d3de019456657b5f: Status 404 returned error can't find the container with id e555fa613c8e8bc8d033fbde7918e568fac0954756db0220d3de019456657b5f Aug 04 19:57:50.425230 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:50.425202 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw"] Aug 04 19:57:50.427657 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:57:50.427626 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode810eb31_7159_4e96_8280_eab12bd4154a.slice/crio-5f0451e3ea92daf9e5536f8ce7b16bf3ab75041befe33d42e19a0c99f7814e3f WatchSource:0}: Error finding container 5f0451e3ea92daf9e5536f8ce7b16bf3ab75041befe33d42e19a0c99f7814e3f: Status 404 returned error can't find the container with id 5f0451e3ea92daf9e5536f8ce7b16bf3ab75041befe33d42e19a0c99f7814e3f Aug 04 19:57:51.309192 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:51.309163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" event={"ID":"dba231ea-db7c-49f6-b435-d87f4457156f","Type":"ContainerStarted","Data":"d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7"} Aug 04 19:57:51.309635 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:51.309219 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" event={"ID":"dba231ea-db7c-49f6-b435-d87f4457156f","Type":"ContainerStarted","Data":"e555fa613c8e8bc8d033fbde7918e568fac0954756db0220d3de019456657b5f"} Aug 04 19:57:51.309635 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:51.309249 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:57:51.310807 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:51.310784 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" event={"ID":"e810eb31-7159-4e96-8280-eab12bd4154a","Type":"ContainerStarted","Data":"45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376"} Aug 04 19:57:51.310937 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:51.310832 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" event={"ID":"e810eb31-7159-4e96-8280-eab12bd4154a","Type":"ContainerStarted","Data":"5f0451e3ea92daf9e5536f8ce7b16bf3ab75041befe33d42e19a0c99f7814e3f"} Aug 04 19:57:51.310937 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:51.310862 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:57:51.324941 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:51.324903 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" podStartSLOduration=2.324890317 podStartE2EDuration="2.324890317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:57:51.323089582 +0000 UTC m=+2516.228690132" watchObservedRunningTime="2026-08-04 19:57:51.324890317 +0000 UTC m=+2516.230490902" Aug 04 19:57:51.338591 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:57:51.338554 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" podStartSLOduration=2.33854204 podStartE2EDuration="2.33854204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:57:51.337543324 +0000 UTC m=+2516.243144055" watchObservedRunningTime="2026-08-04 19:57:51.33854204 +0000 UTC m=+2516.244142579" Aug 04 19:58:02.319940 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:02.319909 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:58:02.322297 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:02.320071 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:58:13.098280 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:13.098249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:14.410613 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:14.410580 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:14.510351 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:14.510315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:17.262346 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:17.262315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:17.360119 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:17.360090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:18.385172 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:18.385137 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:18.534229 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:18.534195 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:18.864845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:18.864816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:18.965004 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:18.964970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:18.966292 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:18.966264 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v"] Aug 04 19:58:18.969677 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:18.969659 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:58:18.972096 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:18.972071 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2e00c3\"" Aug 04 19:58:18.980258 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:18.980237 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v"] Aug 04 19:58:19.002851 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.002821 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774"] Aug 04 19:58:19.006489 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.006452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:58:19.011931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.011907 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2e00c3-dockercfg-s9bl8\"" Aug 04 19:58:19.017776 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.017755 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774"] Aug 04 19:58:19.056045 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.056020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2c35fd-6a91-4966-80d3-d0e0200541da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v\" (UID: \"ae2c35fd-6a91-4966-80d3-d0e0200541da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:58:19.056149 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.056054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhkwc\" (UniqueName: \"kubernetes.io/projected/ae2c35fd-6a91-4966-80d3-d0e0200541da-kube-api-access-qhkwc\") pod \"payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v\" (UID: \"ae2c35fd-6a91-4966-80d3-d0e0200541da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:58:19.157565 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.157487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc86b713-5864-4125-b5e7-e921a981c823-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774\" (UID: \"dc86b713-5864-4125-b5e7-e921a981c823\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:58:19.157565 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.157546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2c35fd-6a91-4966-80d3-d0e0200541da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v\" (UID: \"ae2c35fd-6a91-4966-80d3-d0e0200541da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:58:19.157754 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.157606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhkwc\" (UniqueName: \"kubernetes.io/projected/ae2c35fd-6a91-4966-80d3-d0e0200541da-kube-api-access-qhkwc\") pod \"payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v\" (UID: \"ae2c35fd-6a91-4966-80d3-d0e0200541da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:58:19.157754 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.157636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5v55\" (UniqueName: \"kubernetes.io/projected/dc86b713-5864-4125-b5e7-e921a981c823-kube-api-access-k5v55\") pod \"payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774\" (UID: \"dc86b713-5864-4125-b5e7-e921a981c823\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:58:19.158245 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.158228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2c35fd-6a91-4966-80d3-d0e0200541da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v\" (UID: \"ae2c35fd-6a91-4966-80d3-d0e0200541da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:58:19.165920 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.165881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhkwc\" (UniqueName: \"kubernetes.io/projected/ae2c35fd-6a91-4966-80d3-d0e0200541da-kube-api-access-qhkwc\") pod \"payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v\" (UID: \"ae2c35fd-6a91-4966-80d3-d0e0200541da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:58:19.259023 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.258978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc86b713-5864-4125-b5e7-e921a981c823-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774\" (UID: \"dc86b713-5864-4125-b5e7-e921a981c823\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:58:19.259203 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.259078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5v55\" (UniqueName: \"kubernetes.io/projected/dc86b713-5864-4125-b5e7-e921a981c823-kube-api-access-k5v55\") pod \"payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774\" (UID: \"dc86b713-5864-4125-b5e7-e921a981c823\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:58:19.259692 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.259673 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc86b713-5864-4125-b5e7-e921a981c823-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774\" (UID: \"dc86b713-5864-4125-b5e7-e921a981c823\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:58:19.267953 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.267925 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5v55\" (UniqueName: \"kubernetes.io/projected/dc86b713-5864-4125-b5e7-e921a981c823-kube-api-access-k5v55\") pod \"payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774\" (UID: \"dc86b713-5864-4125-b5e7-e921a981c823\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:58:19.280793 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.280768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:58:19.317179 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.317142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:58:19.429931 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.429903 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v"] Aug 04 19:58:19.432339 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:58:19.432309 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2c35fd_6a91_4966_80d3_d0e0200541da.slice/crio-b23fed20dd0d912e68ee7b26a1ae7bf62224b3b5b0c74aa04dac186813a01e83 WatchSource:0}: Error finding container b23fed20dd0d912e68ee7b26a1ae7bf62224b3b5b0c74aa04dac186813a01e83: Status 404 returned error can't find the container with id b23fed20dd0d912e68ee7b26a1ae7bf62224b3b5b0c74aa04dac186813a01e83 Aug 04 19:58:19.449023 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.448996 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" event={"ID":"ae2c35fd-6a91-4966-80d3-d0e0200541da","Type":"ContainerStarted","Data":"b23fed20dd0d912e68ee7b26a1ae7bf62224b3b5b0c74aa04dac186813a01e83"} Aug 04 19:58:19.463532 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:19.463502 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774"] Aug 04 19:58:19.465657 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:58:19.465630 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc86b713_5864_4125_b5e7_e921a981c823.slice/crio-d97d680abfde8996a2209bfdafeffce0f8db2fe323aa1806c7c0ec8dd305f900 WatchSource:0}: Error finding container d97d680abfde8996a2209bfdafeffce0f8db2fe323aa1806c7c0ec8dd305f900: Status 404 returned error can't find the container with id d97d680abfde8996a2209bfdafeffce0f8db2fe323aa1806c7c0ec8dd305f900 Aug 04 19:58:20.455832 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:20.455790 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" event={"ID":"dc86b713-5864-4125-b5e7-e921a981c823","Type":"ContainerStarted","Data":"01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253"} Aug 04 19:58:20.456274 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:20.455845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" event={"ID":"dc86b713-5864-4125-b5e7-e921a981c823","Type":"ContainerStarted","Data":"d97d680abfde8996a2209bfdafeffce0f8db2fe323aa1806c7c0ec8dd305f900"} Aug 04 19:58:20.456274 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:20.455895 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:58:20.457267 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:20.457242 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" event={"ID":"ae2c35fd-6a91-4966-80d3-d0e0200541da","Type":"ContainerStarted","Data":"d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66"} Aug 04 19:58:20.457413 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:20.457368 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:58:20.477149 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:20.477102 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" podStartSLOduration=2.477082476 podStartE2EDuration="2.477082476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:58:20.474285294 +0000 UTC m=+2545.379885837" watchObservedRunningTime="2026-08-04 19:58:20.477082476 +0000 UTC m=+2545.382683016" Aug 04 19:58:31.464128 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:31.464097 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:58:31.464615 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:31.464164 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:58:31.481501 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:31.481427 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" podStartSLOduration=13.481411043 podStartE2EDuration="13.481411043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:58:20.495880678 +0000 UTC m=+2545.401481216" watchObservedRunningTime="2026-08-04 19:58:31.481411043 +0000 UTC m=+2556.387011581" Aug 04 19:58:42.226328 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:42.226250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:54.652283 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.652223 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:54.677343 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.677314 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw"] Aug 04 19:58:54.677539 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.677512 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" podUID="e810eb31-7159-4e96-8280-eab12bd4154a" containerName="payload-processing" containerID="cri-o://45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376" gracePeriod=30 Aug 04 19:58:54.685593 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.685439 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j"] Aug 04 19:58:54.685920 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.685687 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" podUID="dba231ea-db7c-49f6-b435-d87f4457156f" containerName="payload-pre-processing" containerID="cri-o://d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7" gracePeriod=30 Aug 04 19:58:54.722545 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.722519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:54.778272 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.778248 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:54.886161 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.886133 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:54.929344 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.929320 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:58:54.957071 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.957048 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:58:54.967917 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.967890 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7922\" (UniqueName: \"kubernetes.io/projected/e810eb31-7159-4e96-8280-eab12bd4154a-kube-api-access-b7922\") pod \"e810eb31-7159-4e96-8280-eab12bd4154a\" (UID: \"e810eb31-7159-4e96-8280-eab12bd4154a\") " Aug 04 19:58:54.968054 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.967950 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e810eb31-7159-4e96-8280-eab12bd4154a-plugins-config-volume\") pod \"e810eb31-7159-4e96-8280-eab12bd4154a\" (UID: \"e810eb31-7159-4e96-8280-eab12bd4154a\") " Aug 04 19:58:54.968413 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.968383 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e810eb31-7159-4e96-8280-eab12bd4154a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e810eb31-7159-4e96-8280-eab12bd4154a" (UID: "e810eb31-7159-4e96-8280-eab12bd4154a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:58:54.970305 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:54.970284 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e810eb31-7159-4e96-8280-eab12bd4154a-kube-api-access-b7922" (OuterVolumeSpecName: "kube-api-access-b7922") pod "e810eb31-7159-4e96-8280-eab12bd4154a" (UID: "e810eb31-7159-4e96-8280-eab12bd4154a"). InnerVolumeSpecName "kube-api-access-b7922". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:58:55.069274 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.069248 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wstg\" (UniqueName: \"kubernetes.io/projected/dba231ea-db7c-49f6-b435-d87f4457156f-kube-api-access-6wstg\") pod \"dba231ea-db7c-49f6-b435-d87f4457156f\" (UID: \"dba231ea-db7c-49f6-b435-d87f4457156f\") " Aug 04 19:58:55.069424 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.069304 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dba231ea-db7c-49f6-b435-d87f4457156f-plugins-config-volume\") pod \"dba231ea-db7c-49f6-b435-d87f4457156f\" (UID: \"dba231ea-db7c-49f6-b435-d87f4457156f\") " Aug 04 19:58:55.069592 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.069573 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7922\" (UniqueName: \"kubernetes.io/projected/e810eb31-7159-4e96-8280-eab12bd4154a-kube-api-access-b7922\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:58:55.069648 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.069600 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e810eb31-7159-4e96-8280-eab12bd4154a-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:58:55.069719 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.069699 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba231ea-db7c-49f6-b435-d87f4457156f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dba231ea-db7c-49f6-b435-d87f4457156f" (UID: "dba231ea-db7c-49f6-b435-d87f4457156f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:58:55.071451 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.071432 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba231ea-db7c-49f6-b435-d87f4457156f-kube-api-access-6wstg" (OuterVolumeSpecName: "kube-api-access-6wstg") pod "dba231ea-db7c-49f6-b435-d87f4457156f" (UID: "dba231ea-db7c-49f6-b435-d87f4457156f"). InnerVolumeSpecName "kube-api-access-6wstg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:58:55.168031 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.167948 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:58:55.170084 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.170061 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wstg\" (UniqueName: \"kubernetes.io/projected/dba231ea-db7c-49f6-b435-d87f4457156f-kube-api-access-6wstg\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:58:55.170150 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.170086 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dba231ea-db7c-49f6-b435-d87f4457156f-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:58:55.630793 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.630762 2580 generic.go:356] "Generic (PLEG): container finished" podID="dba231ea-db7c-49f6-b435-d87f4457156f" containerID="d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7" exitCode=0 Aug 04 19:58:55.630908 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.630820 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" Aug 04 19:58:55.630908 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.630844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" event={"ID":"dba231ea-db7c-49f6-b435-d87f4457156f","Type":"ContainerDied","Data":"d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7"} Aug 04 19:58:55.630908 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.630896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j" event={"ID":"dba231ea-db7c-49f6-b435-d87f4457156f","Type":"ContainerDied","Data":"e555fa613c8e8bc8d033fbde7918e568fac0954756db0220d3de019456657b5f"} Aug 04 19:58:55.631050 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.630929 2580 scope.go:117] "RemoveContainer" containerID="d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7" Aug 04 19:58:55.632238 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.632215 2580 generic.go:356] "Generic (PLEG): container finished" podID="e810eb31-7159-4e96-8280-eab12bd4154a" containerID="45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376" exitCode=0 Aug 04 19:58:55.632360 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.632261 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" Aug 04 19:58:55.632360 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.632282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" event={"ID":"e810eb31-7159-4e96-8280-eab12bd4154a","Type":"ContainerDied","Data":"45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376"} Aug 04 19:58:55.632360 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.632315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw" event={"ID":"e810eb31-7159-4e96-8280-eab12bd4154a","Type":"ContainerDied","Data":"5f0451e3ea92daf9e5536f8ce7b16bf3ab75041befe33d42e19a0c99f7814e3f"} Aug 04 19:58:55.639447 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.639431 2580 scope.go:117] "RemoveContainer" containerID="d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7" Aug 04 19:58:55.639729 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:58:55.639710 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7\": container with ID starting with d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7 not found: ID does not exist" containerID="d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7" Aug 04 19:58:55.639791 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.639736 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7"} err="failed to get container status \"d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7\": rpc error: code = NotFound desc = could not find container \"d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7\": container with ID starting with d4fc459b5dd526714935d1bc91706fafde35112967e62f3601704669117b8df7 not found: ID does not exist" Aug 04 19:58:55.639791 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.639753 2580 scope.go:117] "RemoveContainer" containerID="45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376" Aug 04 19:58:55.647519 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.647499 2580 scope.go:117] "RemoveContainer" containerID="45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376" Aug 04 19:58:55.647788 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:58:55.647768 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376\": container with ID starting with 45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376 not found: ID does not exist" containerID="45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376" Aug 04 19:58:55.647880 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.647794 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376"} err="failed to get container status \"45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376\": rpc error: code = NotFound desc = could not find container \"45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376\": container with ID starting with 45e2d301b195a7b8c6436c626f257fa2ae8251d69900549ec1a15266c7cdf376 not found: ID does not exist" Aug 04 19:58:55.649107 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.649087 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw"] Aug 04 19:58:55.654759 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.654740 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4feefd-56cd4cd668-ndvrw"] Aug 04 19:58:55.665111 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.665091 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j"] Aug 04 19:58:55.670937 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:55.670917 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4feefd-5d58667f64-swz7j"] Aug 04 19:58:57.604092 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:57.604062 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba231ea-db7c-49f6-b435-d87f4457156f" path="/var/lib/kubelet/pods/dba231ea-db7c-49f6-b435-d87f4457156f/volumes" Aug 04 19:58:57.604457 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:58:57.604431 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e810eb31-7159-4e96-8280-eab12bd4154a" path="/var/lib/kubelet/pods/e810eb31-7159-4e96-8280-eab12bd4154a/volumes" Aug 04 19:59:01.152829 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:01.152793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:01.215804 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:01.215772 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:01.513070 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:01.513034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:01.692811 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:01.692777 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:17.363751 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.363658 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:17.390868 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.390825 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774"] Aug 04 19:59:17.391090 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.391063 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" podUID="dc86b713-5864-4125-b5e7-e921a981c823" containerName="payload-processing" containerID="cri-o://01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253" gracePeriod=30 Aug 04 19:59:17.401756 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.401728 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v"] Aug 04 19:59:17.401998 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.401961 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" podUID="ae2c35fd-6a91-4966-80d3-d0e0200541da" containerName="payload-pre-processing" containerID="cri-o://d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66" gracePeriod=30 Aug 04 19:59:17.441249 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.441216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:17.498075 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.498049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:17.555938 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.555905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:17.672368 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.672346 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:59:17.701090 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.701063 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:59:17.732763 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.732730 2580 generic.go:356] "Generic (PLEG): container finished" podID="dc86b713-5864-4125-b5e7-e921a981c823" containerID="01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253" exitCode=0 Aug 04 19:59:17.732909 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.732773 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" Aug 04 19:59:17.732909 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.732815 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" event={"ID":"dc86b713-5864-4125-b5e7-e921a981c823","Type":"ContainerDied","Data":"01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253"} Aug 04 19:59:17.732909 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.732888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774" event={"ID":"dc86b713-5864-4125-b5e7-e921a981c823","Type":"ContainerDied","Data":"d97d680abfde8996a2209bfdafeffce0f8db2fe323aa1806c7c0ec8dd305f900"} Aug 04 19:59:17.733076 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.732920 2580 scope.go:117] "RemoveContainer" containerID="01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253" Aug 04 19:59:17.734154 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.734124 2580 generic.go:356] "Generic (PLEG): container finished" podID="ae2c35fd-6a91-4966-80d3-d0e0200541da" containerID="d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66" exitCode=0 Aug 04 19:59:17.734274 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.734162 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" Aug 04 19:59:17.734274 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.734207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" event={"ID":"ae2c35fd-6a91-4966-80d3-d0e0200541da","Type":"ContainerDied","Data":"d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66"} Aug 04 19:59:17.734274 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.734256 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v" event={"ID":"ae2c35fd-6a91-4966-80d3-d0e0200541da","Type":"ContainerDied","Data":"b23fed20dd0d912e68ee7b26a1ae7bf62224b3b5b0c74aa04dac186813a01e83"} Aug 04 19:59:17.741290 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.741273 2580 scope.go:117] "RemoveContainer" containerID="01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253" Aug 04 19:59:17.741581 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:59:17.741561 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253\": container with ID starting with 01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253 not found: ID does not exist" containerID="01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253" Aug 04 19:59:17.741685 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.741588 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253"} err="failed to get container status \"01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253\": rpc error: code = NotFound desc = could not find container \"01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253\": container with ID starting with 01dc7650b02fd0e60da4b5a5877b648e39a249f0efb7cb7df6f67dc606e74253 not found: ID does not exist" Aug 04 19:59:17.741685 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.741604 2580 scope.go:117] "RemoveContainer" containerID="d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66" Aug 04 19:59:17.749941 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.749924 2580 scope.go:117] "RemoveContainer" containerID="d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66" Aug 04 19:59:17.750167 ip-10-0-136-99 kubenswrapper[2580]: E0804 19:59:17.750151 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66\": container with ID starting with d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66 not found: ID does not exist" containerID="d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66" Aug 04 19:59:17.750224 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.750181 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66"} err="failed to get container status \"d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66\": rpc error: code = NotFound desc = could not find container \"d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66\": container with ID starting with d1c3499015d31ba13fd228c97c9363d4380fa20daa5913602ba7aa9aa00eba66 not found: ID does not exist" Aug 04 19:59:17.769495 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.769451 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5v55\" (UniqueName: \"kubernetes.io/projected/dc86b713-5864-4125-b5e7-e921a981c823-kube-api-access-k5v55\") pod \"dc86b713-5864-4125-b5e7-e921a981c823\" (UID: \"dc86b713-5864-4125-b5e7-e921a981c823\") " Aug 04 19:59:17.769631 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.769611 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc86b713-5864-4125-b5e7-e921a981c823-plugins-config-volume\") pod \"dc86b713-5864-4125-b5e7-e921a981c823\" (UID: \"dc86b713-5864-4125-b5e7-e921a981c823\") " Aug 04 19:59:17.769963 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.769942 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc86b713-5864-4125-b5e7-e921a981c823-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dc86b713-5864-4125-b5e7-e921a981c823" (UID: "dc86b713-5864-4125-b5e7-e921a981c823"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:59:17.771654 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.771635 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc86b713-5864-4125-b5e7-e921a981c823-kube-api-access-k5v55" (OuterVolumeSpecName: "kube-api-access-k5v55") pod "dc86b713-5864-4125-b5e7-e921a981c823" (UID: "dc86b713-5864-4125-b5e7-e921a981c823"). InnerVolumeSpecName "kube-api-access-k5v55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:59:17.870595 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.870536 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2c35fd-6a91-4966-80d3-d0e0200541da-plugins-config-volume\") pod \"ae2c35fd-6a91-4966-80d3-d0e0200541da\" (UID: \"ae2c35fd-6a91-4966-80d3-d0e0200541da\") " Aug 04 19:59:17.870595 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.870591 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhkwc\" (UniqueName: \"kubernetes.io/projected/ae2c35fd-6a91-4966-80d3-d0e0200541da-kube-api-access-qhkwc\") pod \"ae2c35fd-6a91-4966-80d3-d0e0200541da\" (UID: \"ae2c35fd-6a91-4966-80d3-d0e0200541da\") " Aug 04 19:59:17.870765 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.870733 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc86b713-5864-4125-b5e7-e921a981c823-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:59:17.870765 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.870744 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5v55\" (UniqueName: \"kubernetes.io/projected/dc86b713-5864-4125-b5e7-e921a981c823-kube-api-access-k5v55\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:59:17.870971 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.870948 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2c35fd-6a91-4966-80d3-d0e0200541da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae2c35fd-6a91-4966-80d3-d0e0200541da" (UID: "ae2c35fd-6a91-4966-80d3-d0e0200541da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:59:17.872673 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.872653 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2c35fd-6a91-4966-80d3-d0e0200541da-kube-api-access-qhkwc" (OuterVolumeSpecName: "kube-api-access-qhkwc") pod "ae2c35fd-6a91-4966-80d3-d0e0200541da" (UID: "ae2c35fd-6a91-4966-80d3-d0e0200541da"). InnerVolumeSpecName "kube-api-access-qhkwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:59:17.971435 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.971406 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2c35fd-6a91-4966-80d3-d0e0200541da-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:59:17.971435 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.971429 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhkwc\" (UniqueName: \"kubernetes.io/projected/ae2c35fd-6a91-4966-80d3-d0e0200541da-kube-api-access-qhkwc\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 19:59:17.987099 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:17.987076 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:18.056555 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:18.056529 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v"] Aug 04 19:59:18.062345 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:18.062320 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e00c3-77f75f44bd-mp22v"] Aug 04 19:59:18.072705 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:18.072683 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774"] Aug 04 19:59:18.079173 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:18.079151 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e00c3-7ddd677fb4-rc774"] Aug 04 19:59:19.603084 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:19.603050 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae2c35fd-6a91-4966-80d3-d0e0200541da" path="/var/lib/kubelet/pods/ae2c35fd-6a91-4966-80d3-d0e0200541da/volumes" Aug 04 19:59:19.603448 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:19.603399 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc86b713-5864-4125-b5e7-e921a981c823" path="/var/lib/kubelet/pods/dc86b713-5864-4125-b5e7-e921a981c823/volumes" Aug 04 19:59:23.716724 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:23.716688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:23.980044 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:23.979965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:24.080621 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:24.080584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:24.253649 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:24.253558 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:24.564896 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:24.564819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:24.923342 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:24.923266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:25.238290 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:25.238258 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:25.505101 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:25.505019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:28.722508 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:28.722458 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:28.825359 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:28.825326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:29.224008 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:29.223981 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:29.950323 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:29.950267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:30.412257 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.412161 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:30.509797 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.509762 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:30.610065 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.610031 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:30.624845 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.624819 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp"] Aug 04 19:59:30.625201 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625189 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba231ea-db7c-49f6-b435-d87f4457156f" containerName="payload-pre-processing" Aug 04 19:59:30.625254 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625203 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba231ea-db7c-49f6-b435-d87f4457156f" containerName="payload-pre-processing" Aug 04 19:59:30.625254 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625213 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc86b713-5864-4125-b5e7-e921a981c823" containerName="payload-processing" Aug 04 19:59:30.625254 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625219 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc86b713-5864-4125-b5e7-e921a981c823" containerName="payload-processing" Aug 04 19:59:30.625254 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625232 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e810eb31-7159-4e96-8280-eab12bd4154a" containerName="payload-processing" Aug 04 19:59:30.625254 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625237 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e810eb31-7159-4e96-8280-eab12bd4154a" containerName="payload-processing" Aug 04 19:59:30.625254 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625243 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2c35fd-6a91-4966-80d3-d0e0200541da" containerName="payload-pre-processing" Aug 04 19:59:30.625254 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625249 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2c35fd-6a91-4966-80d3-d0e0200541da" containerName="payload-pre-processing" Aug 04 19:59:30.625529 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625310 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba231ea-db7c-49f6-b435-d87f4457156f" containerName="payload-pre-processing" Aug 04 19:59:30.625529 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625317 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e810eb31-7159-4e96-8280-eab12bd4154a" containerName="payload-processing" Aug 04 19:59:30.625529 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625324 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae2c35fd-6a91-4966-80d3-d0e0200541da" containerName="payload-pre-processing" Aug 04 19:59:30.625529 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.625332 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc86b713-5864-4125-b5e7-e921a981c823" containerName="payload-processing" Aug 04 19:59:30.629882 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.629867 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 19:59:30.632624 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.632602 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-03ae76\"" Aug 04 19:59:30.632728 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.632665 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-03ae76-dockercfg-s8x7x\"" Aug 04 19:59:30.641878 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.641847 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp"] Aug 04 19:59:30.681927 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.681857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp\" (UID: \"fe5212b5-bea5-4621-aeb6-e067d49f1fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 19:59:30.681927 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.681904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddn4k\" (UniqueName: \"kubernetes.io/projected/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-kube-api-access-ddn4k\") pod \"payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp\" (UID: \"fe5212b5-bea5-4621-aeb6-e067d49f1fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 19:59:30.713828 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.713796 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m"] Aug 04 19:59:30.717230 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.717215 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 19:59:30.726825 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.726802 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m"] Aug 04 19:59:30.782946 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.782907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp\" (UID: \"fe5212b5-bea5-4621-aeb6-e067d49f1fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 19:59:30.783126 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.782956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddn4k\" (UniqueName: \"kubernetes.io/projected/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-kube-api-access-ddn4k\") pod \"payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp\" (UID: \"fe5212b5-bea5-4621-aeb6-e067d49f1fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 19:59:30.783126 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.783076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5qq\" (UniqueName: \"kubernetes.io/projected/1c215215-804c-4916-9cc7-de4c484138c2-kube-api-access-2f5qq\") pod \"payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m\" (UID: \"1c215215-804c-4916-9cc7-de4c484138c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 19:59:30.783126 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.783123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c215215-804c-4916-9cc7-de4c484138c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m\" (UID: \"1c215215-804c-4916-9cc7-de4c484138c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 19:59:30.783761 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.783740 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp\" (UID: \"fe5212b5-bea5-4621-aeb6-e067d49f1fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 19:59:30.792435 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.792399 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddn4k\" (UniqueName: \"kubernetes.io/projected/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-kube-api-access-ddn4k\") pod \"payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp\" (UID: \"fe5212b5-bea5-4621-aeb6-e067d49f1fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 19:59:30.883903 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.883875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5qq\" (UniqueName: \"kubernetes.io/projected/1c215215-804c-4916-9cc7-de4c484138c2-kube-api-access-2f5qq\") pod \"payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m\" (UID: \"1c215215-804c-4916-9cc7-de4c484138c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 19:59:30.884050 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.883912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c215215-804c-4916-9cc7-de4c484138c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m\" (UID: \"1c215215-804c-4916-9cc7-de4c484138c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 19:59:30.884671 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.884655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c215215-804c-4916-9cc7-de4c484138c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m\" (UID: \"1c215215-804c-4916-9cc7-de4c484138c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 19:59:30.892056 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.892031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5qq\" (UniqueName: \"kubernetes.io/projected/1c215215-804c-4916-9cc7-de4c484138c2-kube-api-access-2f5qq\") pod \"payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m\" (UID: \"1c215215-804c-4916-9cc7-de4c484138c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 19:59:30.941039 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:30.941018 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 19:59:31.030858 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.030824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 19:59:31.092661 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.092621 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp"] Aug 04 19:59:31.095233 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:59:31.095179 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5212b5_bea5_4621_aeb6_e067d49f1fd1.slice/crio-a6ca9afcfbff6f11c445cad15fc5d62d1c5d99a2d98ca601a5d20312a63dda23 WatchSource:0}: Error finding container a6ca9afcfbff6f11c445cad15fc5d62d1c5d99a2d98ca601a5d20312a63dda23: Status 404 returned error can't find the container with id a6ca9afcfbff6f11c445cad15fc5d62d1c5d99a2d98ca601a5d20312a63dda23 Aug 04 19:59:31.189008 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.188981 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m"] Aug 04 19:59:31.192668 ip-10-0-136-99 kubenswrapper[2580]: W0804 19:59:31.192610 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c215215_804c_4916_9cc7_de4c484138c2.slice/crio-04e50a09bccac001ecdc8c1c06a38e6ae03d156322035daa0d7f2772e9756912 WatchSource:0}: Error finding container 04e50a09bccac001ecdc8c1c06a38e6ae03d156322035daa0d7f2772e9756912: Status 404 returned error can't find the container with id 04e50a09bccac001ecdc8c1c06a38e6ae03d156322035daa0d7f2772e9756912 Aug 04 19:59:31.803212 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.803168 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" event={"ID":"1c215215-804c-4916-9cc7-de4c484138c2","Type":"ContainerStarted","Data":"38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6"} Aug 04 19:59:31.803387 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.803341 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 19:59:31.803455 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.803384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" event={"ID":"1c215215-804c-4916-9cc7-de4c484138c2","Type":"ContainerStarted","Data":"04e50a09bccac001ecdc8c1c06a38e6ae03d156322035daa0d7f2772e9756912"} Aug 04 19:59:31.805010 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.804971 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" event={"ID":"fe5212b5-bea5-4621-aeb6-e067d49f1fd1","Type":"ContainerStarted","Data":"eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d"} Aug 04 19:59:31.805010 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.805011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" event={"ID":"fe5212b5-bea5-4621-aeb6-e067d49f1fd1","Type":"ContainerStarted","Data":"a6ca9afcfbff6f11c445cad15fc5d62d1c5d99a2d98ca601a5d20312a63dda23"} Aug 04 19:59:31.805210 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.805045 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 19:59:31.827854 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.827808 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" podStartSLOduration=1.8277949919999998 podStartE2EDuration="1.827794992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:59:31.8268881 +0000 UTC m=+2616.732488663" watchObservedRunningTime="2026-08-04 19:59:31.827794992 +0000 UTC m=+2616.733395582" Aug 04 19:59:31.851427 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:31.851385 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" podStartSLOduration=1.851372663 podStartE2EDuration="1.851372663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:59:31.848742177 +0000 UTC m=+2616.754342716" watchObservedRunningTime="2026-08-04 19:59:31.851372663 +0000 UTC m=+2616.756973201" Aug 04 19:59:42.811893 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:42.811860 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 19:59:42.812521 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:42.812496 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 19:59:53.705622 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:53.705585 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:54.046662 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:54.046588 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:54.363815 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:54.363743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:54.954528 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:54.954442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:57.766284 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:57.766249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:57.871564 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:57.871522 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:59.034046 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.034012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:59.287959 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.287887 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:59.483512 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.483477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:59.581645 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.581564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 19:59:59.595674 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.595649 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr"] Aug 04 19:59:59.601096 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.601078 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 19:59:59.604888 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.604868 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-15a874-dockercfg-p4m56\"" Aug 04 19:59:59.604997 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.604869 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-15a874\"" Aug 04 19:59:59.610109 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.610082 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr"] Aug 04 19:59:59.634550 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.634524 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8"] Aug 04 19:59:59.638247 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.638224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 19:59:59.649417 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.649389 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8"] Aug 04 19:59:59.739240 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.739209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790e1565-3bae-43c2-837f-2364a0a0c875-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8\" (UID: \"790e1565-3bae-43c2-837f-2364a0a0c875\") " pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 19:59:59.739418 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.739257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp4nz\" (UniqueName: \"kubernetes.io/projected/42ed2bee-87be-44c4-87c6-351d3a71a23b-kube-api-access-wp4nz\") pod \"payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr\" (UID: \"42ed2bee-87be-44c4-87c6-351d3a71a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 19:59:59.739418 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.739369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ed2bee-87be-44c4-87c6-351d3a71a23b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr\" (UID: \"42ed2bee-87be-44c4-87c6-351d3a71a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 19:59:59.739557 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.739432 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rq2\" (UniqueName: \"kubernetes.io/projected/790e1565-3bae-43c2-837f-2364a0a0c875-kube-api-access-z2rq2\") pod \"payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8\" (UID: \"790e1565-3bae-43c2-837f-2364a0a0c875\") " pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 19:59:59.840392 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.840312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rq2\" (UniqueName: \"kubernetes.io/projected/790e1565-3bae-43c2-837f-2364a0a0c875-kube-api-access-z2rq2\") pod \"payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8\" (UID: \"790e1565-3bae-43c2-837f-2364a0a0c875\") " pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 19:59:59.840591 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.840368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790e1565-3bae-43c2-837f-2364a0a0c875-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8\" (UID: \"790e1565-3bae-43c2-837f-2364a0a0c875\") " pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 19:59:59.840591 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.840562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp4nz\" (UniqueName: \"kubernetes.io/projected/42ed2bee-87be-44c4-87c6-351d3a71a23b-kube-api-access-wp4nz\") pod \"payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr\" (UID: \"42ed2bee-87be-44c4-87c6-351d3a71a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 19:59:59.840731 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.840685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ed2bee-87be-44c4-87c6-351d3a71a23b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr\" (UID: \"42ed2bee-87be-44c4-87c6-351d3a71a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 19:59:59.841143 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.841124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790e1565-3bae-43c2-837f-2364a0a0c875-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8\" (UID: \"790e1565-3bae-43c2-837f-2364a0a0c875\") " pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 19:59:59.841314 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.841295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ed2bee-87be-44c4-87c6-351d3a71a23b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr\" (UID: \"42ed2bee-87be-44c4-87c6-351d3a71a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 19:59:59.849268 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.849241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp4nz\" (UniqueName: \"kubernetes.io/projected/42ed2bee-87be-44c4-87c6-351d3a71a23b-kube-api-access-wp4nz\") pod \"payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr\" (UID: \"42ed2bee-87be-44c4-87c6-351d3a71a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 19:59:59.849365 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.849313 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rq2\" (UniqueName: \"kubernetes.io/projected/790e1565-3bae-43c2-837f-2364a0a0c875-kube-api-access-z2rq2\") pod \"payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8\" (UID: \"790e1565-3bae-43c2-837f-2364a0a0c875\") " pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 19:59:59.912115 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.912085 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 19:59:59.948434 ip-10-0-136-99 kubenswrapper[2580]: I0804 19:59:59.948402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 20:00:00.071769 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:00.071574 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr"] Aug 04 20:00:00.074578 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:00:00.074546 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ed2bee_87be_44c4_87c6_351d3a71a23b.slice/crio-bd34f6a53b81b8bafdbd8c6e4990cad075bd8ed7fabda1e76f113f858239aabb WatchSource:0}: Error finding container bd34f6a53b81b8bafdbd8c6e4990cad075bd8ed7fabda1e76f113f858239aabb: Status 404 returned error can't find the container with id bd34f6a53b81b8bafdbd8c6e4990cad075bd8ed7fabda1e76f113f858239aabb Aug 04 20:00:00.102282 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:00.102255 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8"] Aug 04 20:00:00.110664 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:00:00.110634 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790e1565_3bae_43c2_837f_2364a0a0c875.slice/crio-6397fb184e050a4cc66544ee861b90034f7fe7c463686212df65cf758844855c WatchSource:0}: Error finding container 6397fb184e050a4cc66544ee861b90034f7fe7c463686212df65cf758844855c: Status 404 returned error can't find the container with id 6397fb184e050a4cc66544ee861b90034f7fe7c463686212df65cf758844855c Aug 04 20:00:00.952206 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:00.952172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" event={"ID":"790e1565-3bae-43c2-837f-2364a0a0c875","Type":"ContainerStarted","Data":"55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7"} Aug 04 20:00:00.952371 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:00.952229 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" event={"ID":"790e1565-3bae-43c2-837f-2364a0a0c875","Type":"ContainerStarted","Data":"6397fb184e050a4cc66544ee861b90034f7fe7c463686212df65cf758844855c"} Aug 04 20:00:00.952371 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:00.952262 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 20:00:00.953664 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:00.953640 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" event={"ID":"42ed2bee-87be-44c4-87c6-351d3a71a23b","Type":"ContainerStarted","Data":"00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a"} Aug 04 20:00:00.953740 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:00.953686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" event={"ID":"42ed2bee-87be-44c4-87c6-351d3a71a23b","Type":"ContainerStarted","Data":"bd34f6a53b81b8bafdbd8c6e4990cad075bd8ed7fabda1e76f113f858239aabb"} Aug 04 20:00:00.953740 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:00.953708 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 20:00:00.970785 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:00.970739 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" podStartSLOduration=1.970725355 podStartE2EDuration="1.970725355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:00:00.969006534 +0000 UTC m=+2645.874607072" watchObservedRunningTime="2026-08-04 20:00:00.970725355 +0000 UTC m=+2645.876325894" Aug 04 20:00:00.984555 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:00.984511 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" podStartSLOduration=1.984497128 podStartE2EDuration="1.984497128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:00:00.98369814 +0000 UTC m=+2645.889298690" watchObservedRunningTime="2026-08-04 20:00:00.984497128 +0000 UTC m=+2645.890097667" Aug 04 20:00:11.962082 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:11.962051 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 20:00:11.962541 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:11.962249 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 20:00:22.811677 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:22.811643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:00:33.632609 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:33.632577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:00:55.696494 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:55.696404 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 20:00:55.704505 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:00:55.704483 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 20:01:22.118941 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.118910 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5"] Aug 04 20:01:22.122557 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.122540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.127979 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.127957 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"default-dockercfg-8c6w4\"" Aug 04 20:01:22.128118 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.127986 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"kube-root-ca.crt\"" Aug 04 20:01:22.128118 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.127957 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"auth-test-model-15a874-kserve-self-signed-certs\"" Aug 04 20:01:22.128118 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.128033 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"openshift-service-ca.crt\"" Aug 04 20:01:22.133132 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.133110 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5"] Aug 04 20:01:22.148655 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.148633 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:01:22.234529 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.234496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-model-cache\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.234698 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.234544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd548\" (UniqueName: \"kubernetes.io/projected/b437b892-916f-4ad5-8f0f-b34e6a536d03-kube-api-access-rd548\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.234698 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.234573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-home\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.234698 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.234640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b437b892-916f-4ad5-8f0f-b34e6a536d03-tls-certs\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.234838 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.234692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-tmp-dir\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.234838 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.234729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-dshm\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.336129 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.336101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd548\" (UniqueName: \"kubernetes.io/projected/b437b892-916f-4ad5-8f0f-b34e6a536d03-kube-api-access-rd548\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.336129 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.336130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-home\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.336356 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.336172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b437b892-916f-4ad5-8f0f-b34e6a536d03-tls-certs\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.336356 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.336194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-tmp-dir\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.336356 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.336308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-dshm\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.336542 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.336394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-model-cache\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.336600 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.336583 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-tmp-dir\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.336658 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.336635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-home\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.336737 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.336722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-model-cache\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.338687 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.338667 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-dshm\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.338970 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.338950 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b437b892-916f-4ad5-8f0f-b34e6a536d03-tls-certs\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.346948 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.346919 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd548\" (UniqueName: \"kubernetes.io/projected/b437b892-916f-4ad5-8f0f-b34e6a536d03-kube-api-access-rd548\") pod \"auth-test-model-15a874-kserve-67dc5b8b47-vcnl5\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.431675 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.431589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:22.566480 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:22.566441 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5"] Aug 04 20:01:22.568571 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:01:22.568543 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb437b892_916f_4ad5_8f0f_b34e6a536d03.slice/crio-412301864df8a40c8616d47643d399d4213d4a08cd34a21edb6dc2f6b94f3d79 WatchSource:0}: Error finding container 412301864df8a40c8616d47643d399d4213d4a08cd34a21edb6dc2f6b94f3d79: Status 404 returned error can't find the container with id 412301864df8a40c8616d47643d399d4213d4a08cd34a21edb6dc2f6b94f3d79 Aug 04 20:01:23.355919 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:23.355877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" event={"ID":"b437b892-916f-4ad5-8f0f-b34e6a536d03","Type":"ContainerStarted","Data":"959132d7212a7d03d93c2fc6901c776704d36eac5ed7b8f27f3f9eaab2580c3d"} Aug 04 20:01:23.355919 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:23.355921 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" event={"ID":"b437b892-916f-4ad5-8f0f-b34e6a536d03","Type":"ContainerStarted","Data":"412301864df8a40c8616d47643d399d4213d4a08cd34a21edb6dc2f6b94f3d79"} Aug 04 20:01:23.384876 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:23.384831 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" podStartSLOduration=1.384815722 podStartE2EDuration="1.384815722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:01:23.383418279 +0000 UTC m=+2728.289018817" watchObservedRunningTime="2026-08-04 20:01:23.384815722 +0000 UTC m=+2728.290416261" Aug 04 20:01:32.432065 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:32.432031 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:32.432065 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:32.432074 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:32.445194 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:32.445171 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:33.414284 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:33.414254 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:01:44.099061 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:01:44.099032 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:02:04.411092 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:02:04.411059 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:04:05.910596 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:04:05.910507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:04:22.692528 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:04:22.692494 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:05:46.597910 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:05:46.597869 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:05:47.480449 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:05:47.480412 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5"] Aug 04 20:05:47.495098 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:05:47.495056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:05:55.726856 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:05:55.726833 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 20:05:55.735349 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:05:55.735331 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 20:06:02.500374 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.500331 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" podUID="b437b892-916f-4ad5-8f0f-b34e6a536d03" containerName="main" containerID="cri-o://959132d7212a7d03d93c2fc6901c776704d36eac5ed7b8f27f3f9eaab2580c3d" gracePeriod=45 Aug 04 20:06:02.691698 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.691661 2580 generic.go:356] "Generic (PLEG): container finished" podID="b437b892-916f-4ad5-8f0f-b34e6a536d03" containerID="959132d7212a7d03d93c2fc6901c776704d36eac5ed7b8f27f3f9eaab2580c3d" exitCode=0 Aug 04 20:06:02.691851 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.691732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" event={"ID":"b437b892-916f-4ad5-8f0f-b34e6a536d03","Type":"ContainerDied","Data":"959132d7212a7d03d93c2fc6901c776704d36eac5ed7b8f27f3f9eaab2580c3d"} Aug 04 20:06:02.752245 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.752186 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:06:02.881656 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.881623 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-tmp-dir\") pod \"b437b892-916f-4ad5-8f0f-b34e6a536d03\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " Aug 04 20:06:02.881831 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.881685 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-dshm\") pod \"b437b892-916f-4ad5-8f0f-b34e6a536d03\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " Aug 04 20:06:02.881831 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.881718 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-home\") pod \"b437b892-916f-4ad5-8f0f-b34e6a536d03\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " Aug 04 20:06:02.881831 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.881750 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-model-cache\") pod \"b437b892-916f-4ad5-8f0f-b34e6a536d03\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " Aug 04 20:06:02.881831 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.881771 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b437b892-916f-4ad5-8f0f-b34e6a536d03-tls-certs\") pod \"b437b892-916f-4ad5-8f0f-b34e6a536d03\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " Aug 04 20:06:02.882046 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.881848 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd548\" (UniqueName: \"kubernetes.io/projected/b437b892-916f-4ad5-8f0f-b34e6a536d03-kube-api-access-rd548\") pod \"b437b892-916f-4ad5-8f0f-b34e6a536d03\" (UID: \"b437b892-916f-4ad5-8f0f-b34e6a536d03\") " Aug 04 20:06:02.882046 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.881959 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b437b892-916f-4ad5-8f0f-b34e6a536d03" (UID: "b437b892-916f-4ad5-8f0f-b34e6a536d03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:06:02.882157 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.882037 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-model-cache" (OuterVolumeSpecName: "model-cache") pod "b437b892-916f-4ad5-8f0f-b34e6a536d03" (UID: "b437b892-916f-4ad5-8f0f-b34e6a536d03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:06:02.882157 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.882046 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-home" (OuterVolumeSpecName: "home") pod "b437b892-916f-4ad5-8f0f-b34e6a536d03" (UID: "b437b892-916f-4ad5-8f0f-b34e6a536d03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:06:02.882242 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.882172 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-tmp-dir\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:06:02.882242 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.882183 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-home\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:06:02.882242 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.882192 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-model-cache\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:06:02.884177 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.884149 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-dshm" (OuterVolumeSpecName: "dshm") pod "b437b892-916f-4ad5-8f0f-b34e6a536d03" (UID: "b437b892-916f-4ad5-8f0f-b34e6a536d03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:06:02.884288 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.884202 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b437b892-916f-4ad5-8f0f-b34e6a536d03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b437b892-916f-4ad5-8f0f-b34e6a536d03" (UID: "b437b892-916f-4ad5-8f0f-b34e6a536d03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:06:02.884288 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.884229 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b437b892-916f-4ad5-8f0f-b34e6a536d03-kube-api-access-rd548" (OuterVolumeSpecName: "kube-api-access-rd548") pod "b437b892-916f-4ad5-8f0f-b34e6a536d03" (UID: "b437b892-916f-4ad5-8f0f-b34e6a536d03"). InnerVolumeSpecName "kube-api-access-rd548". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:06:02.982914 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.982873 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b437b892-916f-4ad5-8f0f-b34e6a536d03-dshm\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:06:02.982914 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.982907 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b437b892-916f-4ad5-8f0f-b34e6a536d03-tls-certs\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:06:02.982914 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:02.982919 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd548\" (UniqueName: \"kubernetes.io/projected/b437b892-916f-4ad5-8f0f-b34e6a536d03-kube-api-access-rd548\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:06:03.697888 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:03.697854 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" event={"ID":"b437b892-916f-4ad5-8f0f-b34e6a536d03","Type":"ContainerDied","Data":"412301864df8a40c8616d47643d399d4213d4a08cd34a21edb6dc2f6b94f3d79"} Aug 04 20:06:03.698297 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:03.697906 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5" Aug 04 20:06:03.698297 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:03.697915 2580 scope.go:117] "RemoveContainer" containerID="959132d7212a7d03d93c2fc6901c776704d36eac5ed7b8f27f3f9eaab2580c3d" Aug 04 20:06:03.720705 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:03.720676 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5"] Aug 04 20:06:03.726228 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:03.726195 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/auth-test-model-15a874-kserve-67dc5b8b47-vcnl5"] Aug 04 20:06:04.993548 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:04.993517 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:06:05.603630 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:05.603598 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b437b892-916f-4ad5-8f0f-b34e6a536d03" path="/var/lib/kubelet/pods/b437b892-916f-4ad5-8f0f-b34e6a536d03/volumes" Aug 04 20:06:23.186635 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:23.186603 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:06:23.239654 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:23.239623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:06:38.983439 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:38.983397 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6"] Aug 04 20:06:38.984052 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:38.984034 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b437b892-916f-4ad5-8f0f-b34e6a536d03" containerName="main" Aug 04 20:06:38.984097 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:38.984057 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b437b892-916f-4ad5-8f0f-b34e6a536d03" containerName="main" Aug 04 20:06:38.984178 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:38.984167 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b437b892-916f-4ad5-8f0f-b34e6a536d03" containerName="main" Aug 04 20:06:38.987564 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:38.987546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:38.990939 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:38.990779 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"openshift-service-ca.crt\"" Aug 04 20:06:38.990939 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:38.990821 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"sub-test-model-15a874-kserve-self-signed-certs\"" Aug 04 20:06:38.991120 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:38.991044 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"default-dockercfg-8c6w4\"" Aug 04 20:06:38.991183 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:38.991106 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"kube-root-ca.crt\"" Aug 04 20:06:38.992342 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:38.992316 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6"] Aug 04 20:06:39.001449 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.001428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:06:39.095537 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.095509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tls-certs\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.095670 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.095554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-model-cache\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.095670 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.095590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-dshm\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.095670 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.095624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97btk\" (UniqueName: \"kubernetes.io/projected/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-kube-api-access-97btk\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.095809 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.095704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-home\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.095809 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.095774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tmp-dir\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.197121 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.197099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tmp-dir\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.197252 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.197142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tls-certs\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.197252 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.197164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-model-cache\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.197252 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.197185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-dshm\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.197252 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.197208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97btk\" (UniqueName: \"kubernetes.io/projected/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-kube-api-access-97btk\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.197252 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.197235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-home\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.197602 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.197584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tmp-dir\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.197684 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.197627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-model-cache\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.197684 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.197667 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-home\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.199564 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.199544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-dshm\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.199816 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.199799 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tls-certs\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.205411 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.205387 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97btk\" (UniqueName: \"kubernetes.io/projected/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-kube-api-access-97btk\") pod \"sub-test-model-15a874-kserve-8f88c9bf9-l4lq6\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.298271 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.298188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:39.426893 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.426869 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6"] Aug 04 20:06:39.428659 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:06:39.428622 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb864da6d_b2f7_48dc_84b0_2afcf7ab8721.slice/crio-098b55f669c5433d67218a3e7bd6a807ce33ad110ed166fd34987546cb7ae1b3 WatchSource:0}: Error finding container 098b55f669c5433d67218a3e7bd6a807ce33ad110ed166fd34987546cb7ae1b3: Status 404 returned error can't find the container with id 098b55f669c5433d67218a3e7bd6a807ce33ad110ed166fd34987546cb7ae1b3 Aug 04 20:06:39.873647 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.873613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" event={"ID":"b864da6d-b2f7-48dc-84b0-2afcf7ab8721","Type":"ContainerStarted","Data":"3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac"} Aug 04 20:06:39.873816 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.873660 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" event={"ID":"b864da6d-b2f7-48dc-84b0-2afcf7ab8721","Type":"ContainerStarted","Data":"098b55f669c5433d67218a3e7bd6a807ce33ad110ed166fd34987546cb7ae1b3"} Aug 04 20:06:39.892294 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:39.892248 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" podStartSLOduration=1.8922337649999998 podStartE2EDuration="1.892233765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:06:39.890284271 +0000 UTC m=+3044.795884812" watchObservedRunningTime="2026-08-04 20:06:39.892233765 +0000 UTC m=+3044.797834304" Aug 04 20:06:49.298498 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:49.298392 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:49.298498 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:49.298455 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:49.311210 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:49.311181 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:49.930834 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:49.930808 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:06:53.575139 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:06:53.575102 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:07:53.587583 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:07:53.587548 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:07:54.474620 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:07:54.474588 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6"] Aug 04 20:07:54.487801 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:07:54.487759 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:08:04.387609 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.387578 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5"] Aug 04 20:08:04.391249 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.391230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.395083 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.395062 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"kube-root-ca.crt\"" Aug 04 20:08:04.395221 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.395085 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"default-dockercfg-z7f97\"" Aug 04 20:08:04.395221 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.395113 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"openshift-service-ca.crt\"" Aug 04 20:08:04.395340 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.395225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"rate-test-model-03ae76-kserve-self-signed-certs\"" Aug 04 20:08:04.400720 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.400699 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5"] Aug 04 20:08:04.408654 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.408628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:08:04.559972 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.559937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80902853-bda0-40e0-a8c0-0b1396efc028-tls-certs\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.559972 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.559978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-home\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.560200 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.559995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-dshm\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.560200 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.560032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-tmp-dir\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.560200 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.560072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-model-cache\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.560200 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.560108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btrvg\" (UniqueName: \"kubernetes.io/projected/80902853-bda0-40e0-a8c0-0b1396efc028-kube-api-access-btrvg\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.661092 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.661011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-model-cache\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.661092 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.661062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btrvg\" (UniqueName: \"kubernetes.io/projected/80902853-bda0-40e0-a8c0-0b1396efc028-kube-api-access-btrvg\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.661322 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.661109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80902853-bda0-40e0-a8c0-0b1396efc028-tls-certs\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.661322 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.661136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-home\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.661322 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.661151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-dshm\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.661322 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.661167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-tmp-dir\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.661579 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.661448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-model-cache\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.661579 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.661495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-tmp-dir\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.661579 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.661552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-home\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.663635 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.663608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-dshm\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.663852 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.663837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80902853-bda0-40e0-a8c0-0b1396efc028-tls-certs\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.674795 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.674769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btrvg\" (UniqueName: \"kubernetes.io/projected/80902853-bda0-40e0-a8c0-0b1396efc028-kube-api-access-btrvg\") pod \"rate-test-model-03ae76-kserve-5fd7896c79-fwzl5\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.701316 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.701290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:04.841303 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:04.841271 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5"] Aug 04 20:08:04.843030 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:08:04.843005 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80902853_bda0_40e0_a8c0_0b1396efc028.slice/crio-480efe54fec4da937b576137ae6ef9eccdfd502b5ba218dfc0b2dcf46aa346b8 WatchSource:0}: Error finding container 480efe54fec4da937b576137ae6ef9eccdfd502b5ba218dfc0b2dcf46aa346b8: Status 404 returned error can't find the container with id 480efe54fec4da937b576137ae6ef9eccdfd502b5ba218dfc0b2dcf46aa346b8 Aug 04 20:08:05.283330 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:05.283295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" event={"ID":"80902853-bda0-40e0-a8c0-0b1396efc028","Type":"ContainerStarted","Data":"4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe"} Aug 04 20:08:05.283519 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:05.283340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" event={"ID":"80902853-bda0-40e0-a8c0-0b1396efc028","Type":"ContainerStarted","Data":"480efe54fec4da937b576137ae6ef9eccdfd502b5ba218dfc0b2dcf46aa346b8"} Aug 04 20:08:05.300104 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:05.300056 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" podStartSLOduration=1.3000397399999999 podStartE2EDuration="1.30003974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:08:05.299677017 +0000 UTC m=+3130.205277559" watchObservedRunningTime="2026-08-04 20:08:05.30003974 +0000 UTC m=+3130.205640278" Aug 04 20:08:09.493855 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.493806 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" podUID="b864da6d-b2f7-48dc-84b0-2afcf7ab8721" containerName="main" containerID="cri-o://3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac" gracePeriod=45 Aug 04 20:08:09.748616 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.748551 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:08:09.811922 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.811890 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97btk\" (UniqueName: \"kubernetes.io/projected/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-kube-api-access-97btk\") pod \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " Aug 04 20:08:09.812091 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.811980 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-home\") pod \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " Aug 04 20:08:09.812151 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.812135 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tls-certs\") pod \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " Aug 04 20:08:09.812202 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.812168 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-dshm\") pod \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " Aug 04 20:08:09.812202 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.812195 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-model-cache\") pod \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " Aug 04 20:08:09.812309 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.812210 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tmp-dir\") pod \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\" (UID: \"b864da6d-b2f7-48dc-84b0-2afcf7ab8721\") " Aug 04 20:08:09.812309 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.812215 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-home" (OuterVolumeSpecName: "home") pod "b864da6d-b2f7-48dc-84b0-2afcf7ab8721" (UID: "b864da6d-b2f7-48dc-84b0-2afcf7ab8721"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:08:09.812516 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.812490 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b864da6d-b2f7-48dc-84b0-2afcf7ab8721" (UID: "b864da6d-b2f7-48dc-84b0-2afcf7ab8721"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:08:09.812618 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.812507 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-model-cache" (OuterVolumeSpecName: "model-cache") pod "b864da6d-b2f7-48dc-84b0-2afcf7ab8721" (UID: "b864da6d-b2f7-48dc-84b0-2afcf7ab8721"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:08:09.812681 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.812661 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-home\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.812751 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.812690 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-model-cache\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.812862 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.812807 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tmp-dir\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.814297 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.814267 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-kube-api-access-97btk" (OuterVolumeSpecName: "kube-api-access-97btk") pod "b864da6d-b2f7-48dc-84b0-2afcf7ab8721" (UID: "b864da6d-b2f7-48dc-84b0-2afcf7ab8721"). InnerVolumeSpecName "kube-api-access-97btk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:08:09.814411 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.814390 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-dshm" (OuterVolumeSpecName: "dshm") pod "b864da6d-b2f7-48dc-84b0-2afcf7ab8721" (UID: "b864da6d-b2f7-48dc-84b0-2afcf7ab8721"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:08:09.814564 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.814548 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b864da6d-b2f7-48dc-84b0-2afcf7ab8721" (UID: "b864da6d-b2f7-48dc-84b0-2afcf7ab8721"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:08:09.913453 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.913412 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-tls-certs\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.913453 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.913442 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-dshm\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.913453 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:09.913452 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97btk\" (UniqueName: \"kubernetes.io/projected/b864da6d-b2f7-48dc-84b0-2afcf7ab8721-kube-api-access-97btk\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:08:10.308300 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:10.308267 2580 generic.go:356] "Generic (PLEG): container finished" podID="b864da6d-b2f7-48dc-84b0-2afcf7ab8721" containerID="3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac" exitCode=0 Aug 04 20:08:10.308559 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:10.308319 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" Aug 04 20:08:10.308559 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:10.308335 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" event={"ID":"b864da6d-b2f7-48dc-84b0-2afcf7ab8721","Type":"ContainerDied","Data":"3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac"} Aug 04 20:08:10.308559 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:10.308366 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6" event={"ID":"b864da6d-b2f7-48dc-84b0-2afcf7ab8721","Type":"ContainerDied","Data":"098b55f669c5433d67218a3e7bd6a807ce33ad110ed166fd34987546cb7ae1b3"} Aug 04 20:08:10.308559 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:10.308383 2580 scope.go:117] "RemoveContainer" containerID="3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac" Aug 04 20:08:10.320459 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:10.320385 2580 scope.go:117] "RemoveContainer" containerID="3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac" Aug 04 20:08:10.321099 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:08:10.321059 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac\": container with ID starting with 3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac not found: ID does not exist" containerID="3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac" Aug 04 20:08:10.321170 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:10.321107 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac"} err="failed to get container status \"3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac\": rpc error: code = NotFound desc = could not find container \"3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac\": container with ID starting with 3c8a29ac0271a06923bfb67216623095a35e98831dcd925b4b198fb3ffe112ac not found: ID does not exist" Aug 04 20:08:10.330633 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:10.330609 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6"] Aug 04 20:08:10.334302 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:10.334280 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/sub-test-model-15a874-kserve-8f88c9bf9-l4lq6"] Aug 04 20:08:11.604650 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:11.604621 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b864da6d-b2f7-48dc-84b0-2afcf7ab8721" path="/var/lib/kubelet/pods/b864da6d-b2f7-48dc-84b0-2afcf7ab8721/volumes" Aug 04 20:08:14.701949 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:14.701914 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:14.701949 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:14.701956 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:14.714570 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:14.714547 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:15.344559 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:15.344535 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:08:18.037856 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:08:18.037782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:09:41.755141 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:41.755106 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5"] Aug 04 20:09:43.786077 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:43.786044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:09:45.745854 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:45.745818 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:09:49.153250 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:49.153212 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:09:56.774323 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:56.774281 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" podUID="80902853-bda0-40e0-a8c0-0b1396efc028" containerName="main" containerID="cri-o://4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe" gracePeriod=45 Aug 04 20:09:57.020780 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.020759 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:09:57.056800 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.056733 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btrvg\" (UniqueName: \"kubernetes.io/projected/80902853-bda0-40e0-a8c0-0b1396efc028-kube-api-access-btrvg\") pod \"80902853-bda0-40e0-a8c0-0b1396efc028\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " Aug 04 20:09:57.056800 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.056780 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80902853-bda0-40e0-a8c0-0b1396efc028-tls-certs\") pod \"80902853-bda0-40e0-a8c0-0b1396efc028\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " Aug 04 20:09:57.057011 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.056800 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-tmp-dir\") pod \"80902853-bda0-40e0-a8c0-0b1396efc028\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " Aug 04 20:09:57.057011 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.056852 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-home\") pod \"80902853-bda0-40e0-a8c0-0b1396efc028\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " Aug 04 20:09:57.057011 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.056916 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-dshm\") pod \"80902853-bda0-40e0-a8c0-0b1396efc028\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " Aug 04 20:09:57.057011 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.056943 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-model-cache\") pod \"80902853-bda0-40e0-a8c0-0b1396efc028\" (UID: \"80902853-bda0-40e0-a8c0-0b1396efc028\") " Aug 04 20:09:57.057240 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.057103 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80902853-bda0-40e0-a8c0-0b1396efc028" (UID: "80902853-bda0-40e0-a8c0-0b1396efc028"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:09:57.057240 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.057132 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-home" (OuterVolumeSpecName: "home") pod "80902853-bda0-40e0-a8c0-0b1396efc028" (UID: "80902853-bda0-40e0-a8c0-0b1396efc028"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:09:57.057359 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.057285 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-tmp-dir\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:09:57.057359 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.057302 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-home\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:09:57.057359 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.057322 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-model-cache" (OuterVolumeSpecName: "model-cache") pod "80902853-bda0-40e0-a8c0-0b1396efc028" (UID: "80902853-bda0-40e0-a8c0-0b1396efc028"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:09:57.059287 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.059263 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80902853-bda0-40e0-a8c0-0b1396efc028-kube-api-access-btrvg" (OuterVolumeSpecName: "kube-api-access-btrvg") pod "80902853-bda0-40e0-a8c0-0b1396efc028" (UID: "80902853-bda0-40e0-a8c0-0b1396efc028"). InnerVolumeSpecName "kube-api-access-btrvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:09:57.059410 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.059392 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80902853-bda0-40e0-a8c0-0b1396efc028-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80902853-bda0-40e0-a8c0-0b1396efc028" (UID: "80902853-bda0-40e0-a8c0-0b1396efc028"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:09:57.059410 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.059395 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-dshm" (OuterVolumeSpecName: "dshm") pod "80902853-bda0-40e0-a8c0-0b1396efc028" (UID: "80902853-bda0-40e0-a8c0-0b1396efc028"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:09:57.157697 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.157674 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-dshm\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:09:57.157697 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.157695 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80902853-bda0-40e0-a8c0-0b1396efc028-model-cache\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:09:57.157837 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.157706 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btrvg\" (UniqueName: \"kubernetes.io/projected/80902853-bda0-40e0-a8c0-0b1396efc028-kube-api-access-btrvg\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:09:57.157837 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.157716 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80902853-bda0-40e0-a8c0-0b1396efc028-tls-certs\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:09:57.835976 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.835944 2580 generic.go:356] "Generic (PLEG): container finished" podID="80902853-bda0-40e0-a8c0-0b1396efc028" containerID="4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe" exitCode=0 Aug 04 20:09:57.836391 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.835984 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" Aug 04 20:09:57.836391 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.836001 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" event={"ID":"80902853-bda0-40e0-a8c0-0b1396efc028","Type":"ContainerDied","Data":"4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe"} Aug 04 20:09:57.836391 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.836031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5" event={"ID":"80902853-bda0-40e0-a8c0-0b1396efc028","Type":"ContainerDied","Data":"480efe54fec4da937b576137ae6ef9eccdfd502b5ba218dfc0b2dcf46aa346b8"} Aug 04 20:09:57.836391 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.836048 2580 scope.go:117] "RemoveContainer" containerID="4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe" Aug 04 20:09:57.847311 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.847295 2580 scope.go:117] "RemoveContainer" containerID="4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe" Aug 04 20:09:57.847613 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:09:57.847588 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe\": container with ID starting with 4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe not found: ID does not exist" containerID="4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe" Aug 04 20:09:57.847718 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.847618 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe"} err="failed to get container status \"4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe\": rpc error: code = NotFound desc = could not find container \"4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe\": container with ID starting with 4a3232f248c702d11457c87873e8a5563853eee6e8ef08dda1f5dd84571d3afe not found: ID does not exist" Aug 04 20:09:57.852110 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.852091 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5"] Aug 04 20:09:57.860505 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:57.860481 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/rate-test-model-03ae76-kserve-5fd7896c79-fwzl5"] Aug 04 20:09:59.603328 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:09:59.603297 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80902853-bda0-40e0-a8c0-0b1396efc028" path="/var/lib/kubelet/pods/80902853-bda0-40e0-a8c0-0b1396efc028/volumes" Aug 04 20:10:02.317946 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:02.317911 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:02.531383 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:02.531340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:02.679374 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:02.679291 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:08.501835 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:08.501804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:08.602657 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:08.602627 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:09.660366 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:09.660336 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:10.228354 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.228320 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:10.328109 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.328078 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k"] Aug 04 20:10:10.328497 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.328485 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b864da6d-b2f7-48dc-84b0-2afcf7ab8721" containerName="main" Aug 04 20:10:10.328553 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.328499 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b864da6d-b2f7-48dc-84b0-2afcf7ab8721" containerName="main" Aug 04 20:10:10.328553 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.328515 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80902853-bda0-40e0-a8c0-0b1396efc028" containerName="main" Aug 04 20:10:10.328553 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.328521 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80902853-bda0-40e0-a8c0-0b1396efc028" containerName="main" Aug 04 20:10:10.328655 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.328582 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="80902853-bda0-40e0-a8c0-0b1396efc028" containerName="main" Aug 04 20:10:10.328655 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.328594 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b864da6d-b2f7-48dc-84b0-2afcf7ab8721" containerName="main" Aug 04 20:10:10.331226 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.331202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:10:10.331481 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.331446 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:10.333715 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.333694 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-41ab60\"" Aug 04 20:10:10.333820 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.333742 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-41ab60-dockercfg-6hzvk\"" Aug 04 20:10:10.342198 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.342175 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k"] Aug 04 20:10:10.367929 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.367905 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg"] Aug 04 20:10:10.371236 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.371222 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:10:10.380038 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.380020 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg"] Aug 04 20:10:10.476737 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.476704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmgn\" (UniqueName: \"kubernetes.io/projected/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-kube-api-access-6bmgn\") pod \"payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg\" (UID: \"28ac9d63-fb2e-4ee4-9b5c-14169ba59493\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:10:10.476917 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.476760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg\" (UID: \"28ac9d63-fb2e-4ee4-9b5c-14169ba59493\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:10:10.476917 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.476842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k\" (UID: \"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:10:10.476990 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.476919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkhn\" (UniqueName: \"kubernetes.io/projected/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-kube-api-access-grkhn\") pod \"payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k\" (UID: \"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:10:10.578073 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.577993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmgn\" (UniqueName: \"kubernetes.io/projected/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-kube-api-access-6bmgn\") pod \"payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg\" (UID: \"28ac9d63-fb2e-4ee4-9b5c-14169ba59493\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:10:10.578073 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.578043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg\" (UID: \"28ac9d63-fb2e-4ee4-9b5c-14169ba59493\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:10:10.578073 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.578073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k\" (UID: \"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:10:10.578284 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.578097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkhn\" (UniqueName: \"kubernetes.io/projected/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-kube-api-access-grkhn\") pod \"payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k\" (UID: \"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:10:10.578874 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.578856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg\" (UID: \"28ac9d63-fb2e-4ee4-9b5c-14169ba59493\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:10:10.578926 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.578857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k\" (UID: \"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:10:10.590310 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.590280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkhn\" (UniqueName: \"kubernetes.io/projected/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-kube-api-access-grkhn\") pod \"payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k\" (UID: \"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:10:10.590426 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.590345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmgn\" (UniqueName: \"kubernetes.io/projected/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-kube-api-access-6bmgn\") pod \"payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg\" (UID: \"28ac9d63-fb2e-4ee4-9b5c-14169ba59493\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:10:10.643096 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.643070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:10:10.681882 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.681854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:10:10.794636 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.794606 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k"] Aug 04 20:10:10.797645 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:10:10.797613 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98f0398_5e1a_4a89_ad8f_8ba3bb8092a0.slice/crio-0a5100b145288b170227748903dfe8c98b1ed453c57036beababeed20bed5b7d WatchSource:0}: Error finding container 0a5100b145288b170227748903dfe8c98b1ed453c57036beababeed20bed5b7d: Status 404 returned error can't find the container with id 0a5100b145288b170227748903dfe8c98b1ed453c57036beababeed20bed5b7d Aug 04 20:10:10.843973 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.843950 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg"] Aug 04 20:10:10.845630 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:10:10.845599 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ac9d63_fb2e_4ee4_9b5c_14169ba59493.slice/crio-d160a6f63695c960463abfbd24baf0b9cc4adeadc6a22e32f5cf16b5f08b3956 WatchSource:0}: Error finding container d160a6f63695c960463abfbd24baf0b9cc4adeadc6a22e32f5cf16b5f08b3956: Status 404 returned error can't find the container with id d160a6f63695c960463abfbd24baf0b9cc4adeadc6a22e32f5cf16b5f08b3956 Aug 04 20:10:10.903926 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.903899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" event={"ID":"28ac9d63-fb2e-4ee4-9b5c-14169ba59493","Type":"ContainerStarted","Data":"d160a6f63695c960463abfbd24baf0b9cc4adeadc6a22e32f5cf16b5f08b3956"} Aug 04 20:10:10.905723 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.905691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" event={"ID":"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0","Type":"ContainerStarted","Data":"3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f"} Aug 04 20:10:10.905836 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.905732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" event={"ID":"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0","Type":"ContainerStarted","Data":"0a5100b145288b170227748903dfe8c98b1ed453c57036beababeed20bed5b7d"} Aug 04 20:10:10.905836 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.905767 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:10:10.922640 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.922600 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" podStartSLOduration=0.922585073 podStartE2EDuration="922.585073ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:10:10.921161507 +0000 UTC m=+3255.826762050" watchObservedRunningTime="2026-08-04 20:10:10.922585073 +0000 UTC m=+3255.828185612" Aug 04 20:10:10.926663 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:10.926639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:11.912852 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:11.912816 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" event={"ID":"28ac9d63-fb2e-4ee4-9b5c-14169ba59493","Type":"ContainerStarted","Data":"a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9"} Aug 04 20:10:11.928121 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:11.928064 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" podStartSLOduration=1.92804727 podStartE2EDuration="1.92804727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:10:11.927169479 +0000 UTC m=+3256.832770017" watchObservedRunningTime="2026-08-04 20:10:11.92804727 +0000 UTC m=+3256.833647810" Aug 04 20:10:12.917715 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:12.917683 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:10:21.915670 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:21.915641 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:10:23.924501 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:23.924447 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:10:33.787729 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:33.787681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:34.193653 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:34.193621 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:34.507701 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:34.507618 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:35.405920 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:35.405872 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:37.798409 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:37.798369 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:37.893975 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:37.893941 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:38.926117 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:38.926082 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj"] Aug 04 20:10:38.930227 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:38.930206 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:38.934518 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:38.934494 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9cd851-serving-cert\"" Aug 04 20:10:38.940987 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:38.940965 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj"] Aug 04 20:10:38.963647 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:38.963624 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:39.029177 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.029148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.029290 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.029191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.029290 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.029259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtr4l\" (UniqueName: \"kubernetes.io/projected/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-kube-api-access-vtr4l\") pod \"maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.129721 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.129692 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.129879 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.129741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.129879 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.129775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtr4l\" (UniqueName: \"kubernetes.io/projected/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-kube-api-access-vtr4l\") pod \"maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.129879 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:10:39.129853 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-serving-cert: secret "maas-api-e2e-ipp-b-9cd851-serving-cert" not found Aug 04 20:10:39.130057 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:10:39.129944 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-maas-api-tls podName:2c1bd316-14c8-492d-be50-a2fe8ef71f1f nodeName:}" failed. No retries permitted until 2026-08-04 20:10:39.629924104 +0000 UTC m=+3284.535524630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-maas-api-tls") pod "maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" (UID: "2c1bd316-14c8-492d-be50-a2fe8ef71f1f") : secret "maas-api-e2e-ipp-b-9cd851-serving-cert" not found Aug 04 20:10:39.130443 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.130426 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.139552 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.139530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtr4l\" (UniqueName: \"kubernetes.io/projected/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-kube-api-access-vtr4l\") pod \"maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.412127 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.412087 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:39.500601 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.500568 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r"] Aug 04 20:10:39.502257 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.502227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:10:39.504447 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.504432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:10:39.508096 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.507862 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9cd851-dockercfg-ffvrw\"" Aug 04 20:10:39.508096 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.507876 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9cd851\"" Aug 04 20:10:39.510956 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.510934 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r"] Aug 04 20:10:39.534409 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.534382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvbt\" (UniqueName: \"kubernetes.io/projected/40674855-e638-4c83-9a75-a0e29f5cbd19-kube-api-access-5lvbt\") pod \"payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r\" (UID: \"40674855-e638-4c83-9a75-a0e29f5cbd19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:10:39.534600 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.534419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40674855-e638-4c83-9a75-a0e29f5cbd19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r\" (UID: \"40674855-e638-4c83-9a75-a0e29f5cbd19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:10:39.635608 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.635573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvbt\" (UniqueName: \"kubernetes.io/projected/40674855-e638-4c83-9a75-a0e29f5cbd19-kube-api-access-5lvbt\") pod \"payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r\" (UID: \"40674855-e638-4c83-9a75-a0e29f5cbd19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:10:39.635608 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.635610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40674855-e638-4c83-9a75-a0e29f5cbd19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r\" (UID: \"40674855-e638-4c83-9a75-a0e29f5cbd19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:10:39.635844 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.635672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.636516 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.636437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40674855-e638-4c83-9a75-a0e29f5cbd19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r\" (UID: \"40674855-e638-4c83-9a75-a0e29f5cbd19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:10:39.638337 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.638315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.644510 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.644453 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvbt\" (UniqueName: \"kubernetes.io/projected/40674855-e638-4c83-9a75-a0e29f5cbd19-kube-api-access-5lvbt\") pod \"payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r\" (UID: \"40674855-e638-4c83-9a75-a0e29f5cbd19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:10:39.817108 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.817072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:10:39.844037 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.844009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:39.967868 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.967834 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r"] Aug 04 20:10:39.970350 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:10:39.970312 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40674855_e638_4c83_9a75_a0e29f5cbd19.slice/crio-519e21d6f91af7227845a8aed643cbb3d1a2fe3e66aa0681c43677bdaf2740a0 WatchSource:0}: Error finding container 519e21d6f91af7227845a8aed643cbb3d1a2fe3e66aa0681c43677bdaf2740a0: Status 404 returned error can't find the container with id 519e21d6f91af7227845a8aed643cbb3d1a2fe3e66aa0681c43677bdaf2740a0 Aug 04 20:10:39.996958 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:39.996936 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj"] Aug 04 20:10:40.002708 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:10:40.002683 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1bd316_14c8_492d_be50_a2fe8ef71f1f.slice/crio-a4913a0289df3de6d4fb9e2fcdcb3d7bf8213bf6e6846a8686aeae18ac1e96ec WatchSource:0}: Error finding container a4913a0289df3de6d4fb9e2fcdcb3d7bf8213bf6e6846a8686aeae18ac1e96ec: Status 404 returned error can't find the container with id a4913a0289df3de6d4fb9e2fcdcb3d7bf8213bf6e6846a8686aeae18ac1e96ec Aug 04 20:10:40.004327 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:40.004309 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 20:10:40.055623 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:40.055595 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" event={"ID":"2c1bd316-14c8-492d-be50-a2fe8ef71f1f","Type":"ContainerStarted","Data":"a4913a0289df3de6d4fb9e2fcdcb3d7bf8213bf6e6846a8686aeae18ac1e96ec"} Aug 04 20:10:40.057218 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:40.057198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" event={"ID":"40674855-e638-4c83-9a75-a0e29f5cbd19","Type":"ContainerStarted","Data":"88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0"} Aug 04 20:10:40.057310 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:40.057237 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" event={"ID":"40674855-e638-4c83-9a75-a0e29f5cbd19","Type":"ContainerStarted","Data":"519e21d6f91af7227845a8aed643cbb3d1a2fe3e66aa0681c43677bdaf2740a0"} Aug 04 20:10:40.057310 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:40.057284 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:10:40.072003 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:40.071939 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" podStartSLOduration=1.071926112 podStartE2EDuration="1.071926112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:10:40.070692266 +0000 UTC m=+3284.976292804" watchObservedRunningTime="2026-08-04 20:10:40.071926112 +0000 UTC m=+3284.977526650" Aug 04 20:10:41.064648 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:41.064606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" event={"ID":"2c1bd316-14c8-492d-be50-a2fe8ef71f1f","Type":"ContainerStarted","Data":"c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b"} Aug 04 20:10:41.065091 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:41.064751 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:41.081094 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:41.081041 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" podStartSLOduration=2.727470215 podStartE2EDuration="3.081024998s" totalImagesPullingTime="353.554783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:10:41.079237725 +0000 UTC m=+3285.984838267" watchObservedRunningTime="2026-08-04 20:10:41.081024998 +0000 UTC m=+3285.986625544" Aug 04 20:10:47.075842 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:47.075809 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:10:51.067218 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:51.067186 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:10:55.758523 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:55.758485 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 20:10:55.767574 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:10:55.767551 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j26l2_81435689-dc7b-4c48-b8ec-16034b424424/ovn-acl-logging/0.log" Aug 04 20:11:02.709449 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:11:02.709410 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:11:09.638046 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:11:09.638012 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lqqg_5988f76f-b53e-4784-8e0e-3e1599df97e6/payload-processing/0.log" Aug 04 20:11:26.727798 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:11:26.727726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:09.940380 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:09.940352 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg_28ac9d63-fb2e-4ee4-9b5c-14169ba59493/payload-processing/0.log" Aug 04 20:12:10.063020 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:10.062988 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lqqg_5988f76f-b53e-4784-8e0e-3e1599df97e6/payload-processing/0.log" Aug 04 20:12:10.448681 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:10.448648 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:10.752872 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:10.752777 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:10.958799 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:10.958756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:13.307298 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:13.307253 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:13.407313 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:13.407255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:14.464626 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:14.464591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:14.899221 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:14.899136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:15.067805 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:15.067775 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:15.167441 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:15.167358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:18.188537 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:18.188499 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:18.240639 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:18.240606 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:38.257272 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:38.257232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:38.916433 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:38.916402 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh"] Aug 04 20:12:38.919957 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:38.919938 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh" Aug 04 20:12:38.922237 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:38.922220 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 20:12:38.926861 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:38.926836 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh"] Aug 04 20:12:38.937537 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:38.937513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04daea34-3ccc-4399-840f-b429d5a6df66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh\" (UID: \"04daea34-3ccc-4399-840f-b429d5a6df66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh" Aug 04 20:12:39.038206 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:39.038169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04daea34-3ccc-4399-840f-b429d5a6df66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh\" (UID: \"04daea34-3ccc-4399-840f-b429d5a6df66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh" Aug 04 20:12:39.038870 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:39.038848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04daea34-3ccc-4399-840f-b429d5a6df66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh\" (UID: \"04daea34-3ccc-4399-840f-b429d5a6df66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh" Aug 04 20:12:39.233727 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:39.233695 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh" Aug 04 20:12:39.585858 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:39.585831 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh"] Aug 04 20:12:39.588119 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:12:39.588089 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04daea34_3ccc_4399_840f_b429d5a6df66.slice/crio-2e7ceb1cc6865767bf72387e15bafb17426843508aa981bb508a1795fee7319c WatchSource:0}: Error finding container 2e7ceb1cc6865767bf72387e15bafb17426843508aa981bb508a1795fee7319c: Status 404 returned error can't find the container with id 2e7ceb1cc6865767bf72387e15bafb17426843508aa981bb508a1795fee7319c Aug 04 20:12:39.675556 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:39.675514 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh" event={"ID":"04daea34-3ccc-4399-840f-b429d5a6df66","Type":"ContainerStarted","Data":"2e7ceb1cc6865767bf72387e15bafb17426843508aa981bb508a1795fee7319c"} Aug 04 20:12:42.695895 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:42.695862 2580 generic.go:356] "Generic (PLEG): container finished" podID="04daea34-3ccc-4399-840f-b429d5a6df66" containerID="753d7fb62f9a2f1dc2993fb4c7ebff76f46cf3068e70aaf6030a765bd8d20f41" exitCode=0 Aug 04 20:12:42.696262 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:42.695946 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh" event={"ID":"04daea34-3ccc-4399-840f-b429d5a6df66","Type":"ContainerDied","Data":"753d7fb62f9a2f1dc2993fb4c7ebff76f46cf3068e70aaf6030a765bd8d20f41"} Aug 04 20:12:43.839310 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:43.839272 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh" Aug 04 20:12:43.891401 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:43.891373 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04daea34-3ccc-4399-840f-b429d5a6df66-maas-api-service-ca\") pod \"04daea34-3ccc-4399-840f-b429d5a6df66\" (UID: \"04daea34-3ccc-4399-840f-b429d5a6df66\") " Aug 04 20:12:43.891754 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:43.891730 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04daea34-3ccc-4399-840f-b429d5a6df66-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04daea34-3ccc-4399-840f-b429d5a6df66" (UID: "04daea34-3ccc-4399-840f-b429d5a6df66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:12:43.992493 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:43.992376 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04daea34-3ccc-4399-840f-b429d5a6df66-maas-api-service-ca\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:12:44.707920 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:44.707881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh" event={"ID":"04daea34-3ccc-4399-840f-b429d5a6df66","Type":"ContainerDied","Data":"2e7ceb1cc6865767bf72387e15bafb17426843508aa981bb508a1795fee7319c"} Aug 04 20:12:44.707920 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:44.707902 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh" Aug 04 20:12:44.708123 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:44.707930 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e7ceb1cc6865767bf72387e15bafb17426843508aa981bb508a1795fee7319c" Aug 04 20:12:55.082836 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:55.082757 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:55.147264 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:55.147230 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:55.247046 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:55.247014 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:55.271243 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:55.271209 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh"] Aug 04 20:12:55.274130 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:55.274107 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f4e87-vkjjh"] Aug 04 20:12:55.334629 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:55.334572 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:12:55.604185 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:55.604112 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04daea34-3ccc-4399-840f-b429d5a6df66" path="/var/lib/kubelet/pods/04daea34-3ccc-4399-840f-b429d5a6df66/volumes" Aug 04 20:12:55.638022 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:12:55.637996 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:01.411452 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:01.411413 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:01.462147 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:01.462118 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:01.687401 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:01.687326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:01.899970 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:01.899938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:15.769498 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:15.769447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:20.790170 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:20.790136 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg"] Aug 04 20:13:20.790675 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:20.790346 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" podUID="28ac9d63-fb2e-4ee4-9b5c-14169ba59493" containerName="payload-processing" containerID="cri-o://a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9" gracePeriod=30 Aug 04 20:13:20.796104 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:20.796065 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k"] Aug 04 20:13:20.796313 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:20.796286 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" podUID="c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0" containerName="payload-pre-processing" containerID="cri-o://3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f" gracePeriod=30 Aug 04 20:13:20.803082 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:20.803036 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:20.944643 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:20.944606 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:21.080075 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.080050 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:13:21.083572 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.083554 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:13:21.101632 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.101604 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:21.225432 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.225404 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grkhn\" (UniqueName: \"kubernetes.io/projected/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-kube-api-access-grkhn\") pod \"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0\" (UID: \"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0\") " Aug 04 20:13:21.225657 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.225439 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-plugins-config-volume\") pod \"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0\" (UID: \"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0\") " Aug 04 20:13:21.225657 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.225490 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-plugins-config-volume\") pod \"28ac9d63-fb2e-4ee4-9b5c-14169ba59493\" (UID: \"28ac9d63-fb2e-4ee4-9b5c-14169ba59493\") " Aug 04 20:13:21.225657 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.225556 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bmgn\" (UniqueName: \"kubernetes.io/projected/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-kube-api-access-6bmgn\") pod \"28ac9d63-fb2e-4ee4-9b5c-14169ba59493\" (UID: \"28ac9d63-fb2e-4ee4-9b5c-14169ba59493\") " Aug 04 20:13:21.225903 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.225867 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0" (UID: "c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:13:21.225903 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.225882 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28ac9d63-fb2e-4ee4-9b5c-14169ba59493" (UID: "28ac9d63-fb2e-4ee4-9b5c-14169ba59493"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:13:21.227714 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.227698 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-kube-api-access-grkhn" (OuterVolumeSpecName: "kube-api-access-grkhn") pod "c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0" (UID: "c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0"). InnerVolumeSpecName "kube-api-access-grkhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:13:21.227777 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.227711 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-kube-api-access-6bmgn" (OuterVolumeSpecName: "kube-api-access-6bmgn") pod "28ac9d63-fb2e-4ee4-9b5c-14169ba59493" (UID: "28ac9d63-fb2e-4ee4-9b5c-14169ba59493"). InnerVolumeSpecName "kube-api-access-6bmgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:13:21.327076 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.327017 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grkhn\" (UniqueName: \"kubernetes.io/projected/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-kube-api-access-grkhn\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:13:21.327076 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.327039 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:13:21.327076 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.327048 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:13:21.327076 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.327057 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmgn\" (UniqueName: \"kubernetes.io/projected/28ac9d63-fb2e-4ee4-9b5c-14169ba59493-kube-api-access-6bmgn\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:13:21.339038 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.339013 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:21.892850 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.892774 2580 generic.go:356] "Generic (PLEG): container finished" podID="28ac9d63-fb2e-4ee4-9b5c-14169ba59493" containerID="a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9" exitCode=0 Aug 04 20:13:21.892850 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.892821 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" Aug 04 20:13:21.892850 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.892829 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" event={"ID":"28ac9d63-fb2e-4ee4-9b5c-14169ba59493","Type":"ContainerDied","Data":"a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9"} Aug 04 20:13:21.893347 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.892883 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg" event={"ID":"28ac9d63-fb2e-4ee4-9b5c-14169ba59493","Type":"ContainerDied","Data":"d160a6f63695c960463abfbd24baf0b9cc4adeadc6a22e32f5cf16b5f08b3956"} Aug 04 20:13:21.893347 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.892904 2580 scope.go:117] "RemoveContainer" containerID="a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9" Aug 04 20:13:21.894308 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.894284 2580 generic.go:356] "Generic (PLEG): container finished" podID="c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0" containerID="3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f" exitCode=0 Aug 04 20:13:21.894451 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.894322 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" Aug 04 20:13:21.894451 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.894323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" event={"ID":"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0","Type":"ContainerDied","Data":"3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f"} Aug 04 20:13:21.894607 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.894497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k" event={"ID":"c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0","Type":"ContainerDied","Data":"0a5100b145288b170227748903dfe8c98b1ed453c57036beababeed20bed5b7d"} Aug 04 20:13:21.903806 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.903789 2580 scope.go:117] "RemoveContainer" containerID="a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9" Aug 04 20:13:21.904120 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:13:21.904102 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9\": container with ID starting with a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9 not found: ID does not exist" containerID="a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9" Aug 04 20:13:21.904203 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.904132 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9"} err="failed to get container status \"a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9\": rpc error: code = NotFound desc = could not find container \"a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9\": container with ID starting with a360a9502eb60f2412470b44282a6eea198896b2b5fbce99b735af00838d76d9 not found: ID does not exist" Aug 04 20:13:21.904203 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.904153 2580 scope.go:117] "RemoveContainer" containerID="3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f" Aug 04 20:13:21.912629 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.912610 2580 scope.go:117] "RemoveContainer" containerID="3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f" Aug 04 20:13:21.912893 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:13:21.912877 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f\": container with ID starting with 3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f not found: ID does not exist" containerID="3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f" Aug 04 20:13:21.912941 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.912899 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f"} err="failed to get container status \"3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f\": rpc error: code = NotFound desc = could not find container \"3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f\": container with ID starting with 3e4a158d42fcd219d800d413dbfa2c6669d20e0866a1a466a6d5e19586d1164f not found: ID does not exist" Aug 04 20:13:21.919020 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.918992 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k"] Aug 04 20:13:21.940630 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.940605 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41ab60-6dd5f75c88-lrr2k"] Aug 04 20:13:21.952735 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.952713 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg"] Aug 04 20:13:21.961990 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:21.961968 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41ab60-668b97898b-vxnhg"] Aug 04 20:13:23.604147 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:23.604115 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ac9d63-fb2e-4ee4-9b5c-14169ba59493" path="/var/lib/kubelet/pods/28ac9d63-fb2e-4ee4-9b5c-14169ba59493/volumes" Aug 04 20:13:23.604548 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:23.604483 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0" path="/var/lib/kubelet/pods/c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0/volumes" Aug 04 20:13:26.365492 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:26.365447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:13:34.418694 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:13:34.418658 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:02.884876 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:02.884835 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:03.410866 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:03.410831 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:08.928581 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:08.928539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:09.030131 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:09.030098 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:09.200647 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:09.200620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:09.439728 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:09.439695 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:20.648846 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:20.648750 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj"] Aug 04 20:14:20.649313 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:20.649133 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" podUID="2c1bd316-14c8-492d-be50-a2fe8ef71f1f" containerName="maas-api" containerID="cri-o://c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b" gracePeriod=30 Aug 04 20:14:20.675796 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:20.675770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:20.718317 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:20.718287 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r"] Aug 04 20:14:20.718759 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:20.718726 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" podUID="40674855-e638-4c83-9a75-a0e29f5cbd19" containerName="payload-pre-processing" containerID="cri-o://88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0" gracePeriod=30 Aug 04 20:14:20.754788 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:20.754761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:20.855841 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:20.855807 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:20.955559 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:20.955457 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:20.970205 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:20.970181 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:14:21.010114 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.010090 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:14:21.052533 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.052495 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtr4l\" (UniqueName: \"kubernetes.io/projected/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-kube-api-access-vtr4l\") pod \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " Aug 04 20:14:21.052720 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.052572 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40674855-e638-4c83-9a75-a0e29f5cbd19-plugins-config-volume\") pod \"40674855-e638-4c83-9a75-a0e29f5cbd19\" (UID: \"40674855-e638-4c83-9a75-a0e29f5cbd19\") " Aug 04 20:14:21.052720 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.052617 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lvbt\" (UniqueName: \"kubernetes.io/projected/40674855-e638-4c83-9a75-a0e29f5cbd19-kube-api-access-5lvbt\") pod \"40674855-e638-4c83-9a75-a0e29f5cbd19\" (UID: \"40674855-e638-4c83-9a75-a0e29f5cbd19\") " Aug 04 20:14:21.052720 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.052673 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-maas-api-tls\") pod \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " Aug 04 20:14:21.052720 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.052701 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-openshift-service-ca\") pod \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\" (UID: \"2c1bd316-14c8-492d-be50-a2fe8ef71f1f\") " Aug 04 20:14:21.053003 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.052977 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40674855-e638-4c83-9a75-a0e29f5cbd19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40674855-e638-4c83-9a75-a0e29f5cbd19" (UID: "40674855-e638-4c83-9a75-a0e29f5cbd19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:21.053213 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.053179 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c1bd316-14c8-492d-be50-a2fe8ef71f1f" (UID: "2c1bd316-14c8-492d-be50-a2fe8ef71f1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:21.054897 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.054867 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40674855-e638-4c83-9a75-a0e29f5cbd19-kube-api-access-5lvbt" (OuterVolumeSpecName: "kube-api-access-5lvbt") pod "40674855-e638-4c83-9a75-a0e29f5cbd19" (UID: "40674855-e638-4c83-9a75-a0e29f5cbd19"). InnerVolumeSpecName "kube-api-access-5lvbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:21.054998 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.054968 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-kube-api-access-vtr4l" (OuterVolumeSpecName: "kube-api-access-vtr4l") pod "2c1bd316-14c8-492d-be50-a2fe8ef71f1f" (UID: "2c1bd316-14c8-492d-be50-a2fe8ef71f1f"). InnerVolumeSpecName "kube-api-access-vtr4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:21.054998 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.054978 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c1bd316-14c8-492d-be50-a2fe8ef71f1f" (UID: "2c1bd316-14c8-492d-be50-a2fe8ef71f1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:14:21.154256 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.154222 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40674855-e638-4c83-9a75-a0e29f5cbd19-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:21.154256 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.154250 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvbt\" (UniqueName: \"kubernetes.io/projected/40674855-e638-4c83-9a75-a0e29f5cbd19-kube-api-access-5lvbt\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:21.154256 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.154260 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-maas-api-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:21.154538 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.154269 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-openshift-service-ca\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:21.154538 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.154279 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtr4l\" (UniqueName: \"kubernetes.io/projected/2c1bd316-14c8-492d-be50-a2fe8ef71f1f-kube-api-access-vtr4l\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:21.191846 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.191813 2580 generic.go:356] "Generic (PLEG): container finished" podID="2c1bd316-14c8-492d-be50-a2fe8ef71f1f" containerID="c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b" exitCode=0 Aug 04 20:14:21.192007 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.191868 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" Aug 04 20:14:21.192007 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.191897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" event={"ID":"2c1bd316-14c8-492d-be50-a2fe8ef71f1f","Type":"ContainerDied","Data":"c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b"} Aug 04 20:14:21.192007 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.191960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj" event={"ID":"2c1bd316-14c8-492d-be50-a2fe8ef71f1f","Type":"ContainerDied","Data":"a4913a0289df3de6d4fb9e2fcdcb3d7bf8213bf6e6846a8686aeae18ac1e96ec"} Aug 04 20:14:21.192007 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.191984 2580 scope.go:117] "RemoveContainer" containerID="c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b" Aug 04 20:14:21.193348 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.193326 2580 generic.go:356] "Generic (PLEG): container finished" podID="40674855-e638-4c83-9a75-a0e29f5cbd19" containerID="88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0" exitCode=0 Aug 04 20:14:21.193445 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.193370 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" Aug 04 20:14:21.193445 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.193412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" event={"ID":"40674855-e638-4c83-9a75-a0e29f5cbd19","Type":"ContainerDied","Data":"88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0"} Aug 04 20:14:21.193570 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.193495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r" event={"ID":"40674855-e638-4c83-9a75-a0e29f5cbd19","Type":"ContainerDied","Data":"519e21d6f91af7227845a8aed643cbb3d1a2fe3e66aa0681c43677bdaf2740a0"} Aug 04 20:14:21.201346 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.201324 2580 scope.go:117] "RemoveContainer" containerID="c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b" Aug 04 20:14:21.201667 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:14:21.201639 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b\": container with ID starting with c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b not found: ID does not exist" containerID="c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b" Aug 04 20:14:21.201765 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.201677 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b"} err="failed to get container status \"c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b\": rpc error: code = NotFound desc = could not find container \"c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b\": container with ID starting with c650dec7e15f76c88deb1f6b377608fe0ee4488576ee974c8c2954e9c9abd54b not found: ID does not exist" Aug 04 20:14:21.201765 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.201696 2580 scope.go:117] "RemoveContainer" containerID="88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0" Aug 04 20:14:21.210006 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.209987 2580 scope.go:117] "RemoveContainer" containerID="88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0" Aug 04 20:14:21.210247 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:14:21.210225 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0\": container with ID starting with 88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0 not found: ID does not exist" containerID="88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0" Aug 04 20:14:21.210306 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.210257 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0"} err="failed to get container status \"88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0\": rpc error: code = NotFound desc = could not find container \"88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0\": container with ID starting with 88568e56ac798a6407cd5a11e4295dba0d4da79087b79dc014bda3fcb18598c0 not found: ID does not exist" Aug 04 20:14:21.216287 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.216266 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r"] Aug 04 20:14:21.218641 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.218624 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9cd851-6694746744-s8z2r"] Aug 04 20:14:21.227778 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.227752 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj"] Aug 04 20:14:21.235736 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.235704 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9cd851-796b7f88cb-jhhvj"] Aug 04 20:14:21.421548 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.421501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:21.605173 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.605079 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c1bd316-14c8-492d-be50-a2fe8ef71f1f" path="/var/lib/kubelet/pods/2c1bd316-14c8-492d-be50-a2fe8ef71f1f/volumes" Aug 04 20:14:21.605569 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:21.605549 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40674855-e638-4c83-9a75-a0e29f5cbd19" path="/var/lib/kubelet/pods/40674855-e638-4c83-9a75-a0e29f5cbd19/volumes" Aug 04 20:14:27.315736 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:27.315692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:27.376830 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:27.376804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:27.604542 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:27.604393 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:27.780081 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:27.780052 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:38.476766 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.476726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:38.515670 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.515640 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m"] Aug 04 20:14:38.515915 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.515876 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" podUID="1c215215-804c-4916-9cc7-de4c484138c2" containerName="payload-processing" containerID="cri-o://38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6" gracePeriod=30 Aug 04 20:14:38.527347 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.527298 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp"] Aug 04 20:14:38.527588 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.527565 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" podUID="fe5212b5-bea5-4621-aeb6-e067d49f1fd1" containerName="payload-pre-processing" containerID="cri-o://eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d" gracePeriod=30 Aug 04 20:14:38.575308 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.575281 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:38.647404 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.647366 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:38.714106 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.714065 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:38.809187 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.809166 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 20:14:38.812243 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.812223 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 20:14:38.918205 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.918178 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-plugins-config-volume\") pod \"fe5212b5-bea5-4621-aeb6-e067d49f1fd1\" (UID: \"fe5212b5-bea5-4621-aeb6-e067d49f1fd1\") " Aug 04 20:14:38.918367 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.918230 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddn4k\" (UniqueName: \"kubernetes.io/projected/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-kube-api-access-ddn4k\") pod \"fe5212b5-bea5-4621-aeb6-e067d49f1fd1\" (UID: \"fe5212b5-bea5-4621-aeb6-e067d49f1fd1\") " Aug 04 20:14:38.918367 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.918309 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f5qq\" (UniqueName: \"kubernetes.io/projected/1c215215-804c-4916-9cc7-de4c484138c2-kube-api-access-2f5qq\") pod \"1c215215-804c-4916-9cc7-de4c484138c2\" (UID: \"1c215215-804c-4916-9cc7-de4c484138c2\") " Aug 04 20:14:38.918367 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.918324 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c215215-804c-4916-9cc7-de4c484138c2-plugins-config-volume\") pod \"1c215215-804c-4916-9cc7-de4c484138c2\" (UID: \"1c215215-804c-4916-9cc7-de4c484138c2\") " Aug 04 20:14:38.918681 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.918645 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe5212b5-bea5-4621-aeb6-e067d49f1fd1" (UID: "fe5212b5-bea5-4621-aeb6-e067d49f1fd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:38.918801 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.918757 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c215215-804c-4916-9cc7-de4c484138c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c215215-804c-4916-9cc7-de4c484138c2" (UID: "1c215215-804c-4916-9cc7-de4c484138c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:38.920617 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.920592 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c215215-804c-4916-9cc7-de4c484138c2-kube-api-access-2f5qq" (OuterVolumeSpecName: "kube-api-access-2f5qq") pod "1c215215-804c-4916-9cc7-de4c484138c2" (UID: "1c215215-804c-4916-9cc7-de4c484138c2"). InnerVolumeSpecName "kube-api-access-2f5qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:38.920739 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:38.920721 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-kube-api-access-ddn4k" (OuterVolumeSpecName: "kube-api-access-ddn4k") pod "fe5212b5-bea5-4621-aeb6-e067d49f1fd1" (UID: "fe5212b5-bea5-4621-aeb6-e067d49f1fd1"). InnerVolumeSpecName "kube-api-access-ddn4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:39.019831 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.019755 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f5qq\" (UniqueName: \"kubernetes.io/projected/1c215215-804c-4916-9cc7-de4c484138c2-kube-api-access-2f5qq\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:39.019831 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.019779 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c215215-804c-4916-9cc7-de4c484138c2-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:39.019831 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.019789 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:39.019831 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.019797 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddn4k\" (UniqueName: \"kubernetes.io/projected/fe5212b5-bea5-4621-aeb6-e067d49f1fd1-kube-api-access-ddn4k\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:39.036635 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.036607 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:39.299511 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.299382 2580 generic.go:356] "Generic (PLEG): container finished" podID="fe5212b5-bea5-4621-aeb6-e067d49f1fd1" containerID="eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d" exitCode=0 Aug 04 20:14:39.299511 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.299420 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" Aug 04 20:14:39.299511 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.299458 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" event={"ID":"fe5212b5-bea5-4621-aeb6-e067d49f1fd1","Type":"ContainerDied","Data":"eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d"} Aug 04 20:14:39.299797 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.299533 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp" event={"ID":"fe5212b5-bea5-4621-aeb6-e067d49f1fd1","Type":"ContainerDied","Data":"a6ca9afcfbff6f11c445cad15fc5d62d1c5d99a2d98ca601a5d20312a63dda23"} Aug 04 20:14:39.299797 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.299565 2580 scope.go:117] "RemoveContainer" containerID="eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d" Aug 04 20:14:39.301178 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.301058 2580 generic.go:356] "Generic (PLEG): container finished" podID="1c215215-804c-4916-9cc7-de4c484138c2" containerID="38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6" exitCode=0 Aug 04 20:14:39.301178 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.301108 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" event={"ID":"1c215215-804c-4916-9cc7-de4c484138c2","Type":"ContainerDied","Data":"38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6"} Aug 04 20:14:39.301178 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.301114 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" Aug 04 20:14:39.301178 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.301147 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m" event={"ID":"1c215215-804c-4916-9cc7-de4c484138c2","Type":"ContainerDied","Data":"04e50a09bccac001ecdc8c1c06a38e6ae03d156322035daa0d7f2772e9756912"} Aug 04 20:14:39.308556 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.308536 2580 scope.go:117] "RemoveContainer" containerID="eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d" Aug 04 20:14:39.308857 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:14:39.308833 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d\": container with ID starting with eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d not found: ID does not exist" containerID="eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d" Aug 04 20:14:39.308915 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.308872 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d"} err="failed to get container status \"eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d\": rpc error: code = NotFound desc = could not find container \"eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d\": container with ID starting with eb87dbca4e5bec50e085f3aeaaab76a0ef450f2f48942be71bc61dace1aa613d not found: ID does not exist" Aug 04 20:14:39.308915 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.308897 2580 scope.go:117] "RemoveContainer" containerID="38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6" Aug 04 20:14:39.317239 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.317225 2580 scope.go:117] "RemoveContainer" containerID="38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6" Aug 04 20:14:39.317479 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:14:39.317445 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6\": container with ID starting with 38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6 not found: ID does not exist" containerID="38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6" Aug 04 20:14:39.317530 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.317489 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6"} err="failed to get container status \"38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6\": rpc error: code = NotFound desc = could not find container \"38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6\": container with ID starting with 38f774582a637e8e0f8a12a7948fc7bf3bf8aac7e3146bc945308d4f05a9a1f6 not found: ID does not exist" Aug 04 20:14:39.324547 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.324528 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m"] Aug 04 20:14:39.329961 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.329940 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-03ae76-5c9cd7449d-jlv2m"] Aug 04 20:14:39.339764 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.339744 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp"] Aug 04 20:14:39.345374 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.345354 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-03ae76-7d966846f6-w2svp"] Aug 04 20:14:39.603181 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.603113 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c215215-804c-4916-9cc7-de4c484138c2" path="/var/lib/kubelet/pods/1c215215-804c-4916-9cc7-de4c484138c2/volumes" Aug 04 20:14:39.603520 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:39.603476 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5212b5-bea5-4621-aeb6-e067d49f1fd1" path="/var/lib/kubelet/pods/fe5212b5-bea5-4621-aeb6-e067d49f1fd1/volumes" Aug 04 20:14:44.922512 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:44.922459 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:44.986528 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:44.986495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:45.165776 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.165725 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:45.396867 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.396836 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:45.506611 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.506578 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q"] Aug 04 20:14:45.506988 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.506976 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40674855-e638-4c83-9a75-a0e29f5cbd19" containerName="payload-pre-processing" Aug 04 20:14:45.507033 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.506991 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40674855-e638-4c83-9a75-a0e29f5cbd19" containerName="payload-pre-processing" Aug 04 20:14:45.507033 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.506999 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ac9d63-fb2e-4ee4-9b5c-14169ba59493" containerName="payload-processing" Aug 04 20:14:45.507033 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507006 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ac9d63-fb2e-4ee4-9b5c-14169ba59493" containerName="payload-processing" Aug 04 20:14:45.507033 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507017 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c215215-804c-4916-9cc7-de4c484138c2" containerName="payload-processing" Aug 04 20:14:45.507033 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507022 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c215215-804c-4916-9cc7-de4c484138c2" containerName="payload-processing" Aug 04 20:14:45.507033 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507032 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0" containerName="payload-pre-processing" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507037 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0" containerName="payload-pre-processing" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507045 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04daea34-3ccc-4399-840f-b429d5a6df66" containerName="revoke-keys" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507050 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04daea34-3ccc-4399-840f-b429d5a6df66" containerName="revoke-keys" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507059 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1bd316-14c8-492d-be50-a2fe8ef71f1f" containerName="maas-api" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507064 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1bd316-14c8-492d-be50-a2fe8ef71f1f" containerName="maas-api" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507071 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5212b5-bea5-4621-aeb6-e067d49f1fd1" containerName="payload-pre-processing" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507076 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5212b5-bea5-4621-aeb6-e067d49f1fd1" containerName="payload-pre-processing" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507134 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5212b5-bea5-4621-aeb6-e067d49f1fd1" containerName="payload-pre-processing" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507141 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1bd316-14c8-492d-be50-a2fe8ef71f1f" containerName="maas-api" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507149 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c215215-804c-4916-9cc7-de4c484138c2" containerName="payload-processing" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507155 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c98f0398-5e1a-4a89-ad8f-8ba3bb8092a0" containerName="payload-pre-processing" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507159 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ac9d63-fb2e-4ee4-9b5c-14169ba59493" containerName="payload-processing" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507165 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="04daea34-3ccc-4399-840f-b429d5a6df66" containerName="revoke-keys" Aug 04 20:14:45.507215 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.507171 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40674855-e638-4c83-9a75-a0e29f5cbd19" containerName="payload-pre-processing" Aug 04 20:14:45.510136 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.510119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" Aug 04 20:14:45.512710 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.512694 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 20:14:45.517792 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.517760 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q"] Aug 04 20:14:45.578253 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.578223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621d2f19-a164-4f12-bcd1-f8c9ff38b76d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-15a874-zf24q\" (UID: \"621d2f19-a164-4f12-bcd1-f8c9ff38b76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" Aug 04 20:14:45.678772 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.678680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621d2f19-a164-4f12-bcd1-f8c9ff38b76d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-15a874-zf24q\" (UID: \"621d2f19-a164-4f12-bcd1-f8c9ff38b76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" Aug 04 20:14:45.679362 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.679326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621d2f19-a164-4f12-bcd1-f8c9ff38b76d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-15a874-zf24q\" (UID: \"621d2f19-a164-4f12-bcd1-f8c9ff38b76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" Aug 04 20:14:45.824114 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.824081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" Aug 04 20:14:45.958733 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:45.958707 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q"] Aug 04 20:14:45.961154 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:14:45.961121 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621d2f19_a164_4f12_bcd1_f8c9ff38b76d.slice/crio-782f8d6956c122176c5c6267980dd864cecbcc19e06cd6ea86ca1a6dcc1af00d WatchSource:0}: Error finding container 782f8d6956c122176c5c6267980dd864cecbcc19e06cd6ea86ca1a6dcc1af00d: Status 404 returned error can't find the container with id 782f8d6956c122176c5c6267980dd864cecbcc19e06cd6ea86ca1a6dcc1af00d Aug 04 20:14:46.337232 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:46.337200 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" event={"ID":"621d2f19-a164-4f12-bcd1-f8c9ff38b76d","Type":"ContainerStarted","Data":"367d1bc42dc4e7ddaee3250e693328a5db7a3ad559423934321ffb3759730811"} Aug 04 20:14:46.337410 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:46.337252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" event={"ID":"621d2f19-a164-4f12-bcd1-f8c9ff38b76d","Type":"ContainerStarted","Data":"782f8d6956c122176c5c6267980dd864cecbcc19e06cd6ea86ca1a6dcc1af00d"} Aug 04 20:14:46.369813 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:46.369768 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" podStartSLOduration=1.369754173 podStartE2EDuration="1.369754173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:14:46.369072698 +0000 UTC m=+3531.274673238" watchObservedRunningTime="2026-08-04 20:14:46.369754173 +0000 UTC m=+3531.275354711" Aug 04 20:14:47.344723 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:47.344689 2580 generic.go:356] "Generic (PLEG): container finished" podID="621d2f19-a164-4f12-bcd1-f8c9ff38b76d" containerID="367d1bc42dc4e7ddaee3250e693328a5db7a3ad559423934321ffb3759730811" exitCode=0 Aug 04 20:14:47.345121 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:47.344781 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" event={"ID":"621d2f19-a164-4f12-bcd1-f8c9ff38b76d","Type":"ContainerDied","Data":"367d1bc42dc4e7ddaee3250e693328a5db7a3ad559423934321ffb3759730811"} Aug 04 20:14:48.488193 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:48.488170 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" Aug 04 20:14:48.605161 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:48.605098 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621d2f19-a164-4f12-bcd1-f8c9ff38b76d-maas-api-service-ca\") pod \"621d2f19-a164-4f12-bcd1-f8c9ff38b76d\" (UID: \"621d2f19-a164-4f12-bcd1-f8c9ff38b76d\") " Aug 04 20:14:48.605436 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:48.605417 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621d2f19-a164-4f12-bcd1-f8c9ff38b76d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "621d2f19-a164-4f12-bcd1-f8c9ff38b76d" (UID: "621d2f19-a164-4f12-bcd1-f8c9ff38b76d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:48.706947 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:48.706862 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621d2f19-a164-4f12-bcd1-f8c9ff38b76d-maas-api-service-ca\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:49.355269 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.355224 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" event={"ID":"621d2f19-a164-4f12-bcd1-f8c9ff38b76d","Type":"ContainerDied","Data":"782f8d6956c122176c5c6267980dd864cecbcc19e06cd6ea86ca1a6dcc1af00d"} Aug 04 20:14:49.355481 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.355308 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="782f8d6956c122176c5c6267980dd864cecbcc19e06cd6ea86ca1a6dcc1af00d" Aug 04 20:14:49.355481 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.355239 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q" Aug 04 20:14:49.663273 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.663183 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:49.673105 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.673076 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8"] Aug 04 20:14:49.673290 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.673265 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" podUID="790e1565-3bae-43c2-837f-2364a0a0c875" containerName="payload-processing" containerID="cri-o://55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7" gracePeriod=30 Aug 04 20:14:49.682772 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.682741 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr"] Aug 04 20:14:49.683169 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.683137 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" podUID="42ed2bee-87be-44c4-87c6-351d3a71a23b" containerName="payload-pre-processing" containerID="cri-o://00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a" gracePeriod=30 Aug 04 20:14:49.817280 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.817234 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q"] Aug 04 20:14:49.820329 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.820206 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-15a874-zf24q"] Aug 04 20:14:49.957998 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.957944 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 20:14:49.961477 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:49.961442 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 20:14:50.118679 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.118649 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ed2bee-87be-44c4-87c6-351d3a71a23b-plugins-config-volume\") pod \"42ed2bee-87be-44c4-87c6-351d3a71a23b\" (UID: \"42ed2bee-87be-44c4-87c6-351d3a71a23b\") " Aug 04 20:14:50.118837 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.118698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp4nz\" (UniqueName: \"kubernetes.io/projected/42ed2bee-87be-44c4-87c6-351d3a71a23b-kube-api-access-wp4nz\") pod \"42ed2bee-87be-44c4-87c6-351d3a71a23b\" (UID: \"42ed2bee-87be-44c4-87c6-351d3a71a23b\") " Aug 04 20:14:50.118837 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.118729 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790e1565-3bae-43c2-837f-2364a0a0c875-plugins-config-volume\") pod \"790e1565-3bae-43c2-837f-2364a0a0c875\" (UID: \"790e1565-3bae-43c2-837f-2364a0a0c875\") " Aug 04 20:14:50.118837 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.118753 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2rq2\" (UniqueName: \"kubernetes.io/projected/790e1565-3bae-43c2-837f-2364a0a0c875-kube-api-access-z2rq2\") pod \"790e1565-3bae-43c2-837f-2364a0a0c875\" (UID: \"790e1565-3bae-43c2-837f-2364a0a0c875\") " Aug 04 20:14:50.119050 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.119027 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ed2bee-87be-44c4-87c6-351d3a71a23b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42ed2bee-87be-44c4-87c6-351d3a71a23b" (UID: "42ed2bee-87be-44c4-87c6-351d3a71a23b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:50.119174 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.119149 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790e1565-3bae-43c2-837f-2364a0a0c875-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "790e1565-3bae-43c2-837f-2364a0a0c875" (UID: "790e1565-3bae-43c2-837f-2364a0a0c875"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:50.120929 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.120910 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ed2bee-87be-44c4-87c6-351d3a71a23b-kube-api-access-wp4nz" (OuterVolumeSpecName: "kube-api-access-wp4nz") pod "42ed2bee-87be-44c4-87c6-351d3a71a23b" (UID: "42ed2bee-87be-44c4-87c6-351d3a71a23b"). InnerVolumeSpecName "kube-api-access-wp4nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:50.121059 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.121039 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790e1565-3bae-43c2-837f-2364a0a0c875-kube-api-access-z2rq2" (OuterVolumeSpecName: "kube-api-access-z2rq2") pod "790e1565-3bae-43c2-837f-2364a0a0c875" (UID: "790e1565-3bae-43c2-837f-2364a0a0c875"). InnerVolumeSpecName "kube-api-access-z2rq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:50.220183 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.220154 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ed2bee-87be-44c4-87c6-351d3a71a23b-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.220183 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.220179 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp4nz\" (UniqueName: \"kubernetes.io/projected/42ed2bee-87be-44c4-87c6-351d3a71a23b-kube-api-access-wp4nz\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.220339 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.220194 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790e1565-3bae-43c2-837f-2364a0a0c875-plugins-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.220339 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.220206 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2rq2\" (UniqueName: \"kubernetes.io/projected/790e1565-3bae-43c2-837f-2364a0a0c875-kube-api-access-z2rq2\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.360671 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.360633 2580 generic.go:356] "Generic (PLEG): container finished" podID="790e1565-3bae-43c2-837f-2364a0a0c875" containerID="55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7" exitCode=0 Aug 04 20:14:50.360822 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.360685 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" Aug 04 20:14:50.360822 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.360724 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" event={"ID":"790e1565-3bae-43c2-837f-2364a0a0c875","Type":"ContainerDied","Data":"55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7"} Aug 04 20:14:50.360822 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.360813 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8" event={"ID":"790e1565-3bae-43c2-837f-2364a0a0c875","Type":"ContainerDied","Data":"6397fb184e050a4cc66544ee861b90034f7fe7c463686212df65cf758844855c"} Aug 04 20:14:50.361000 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.360839 2580 scope.go:117] "RemoveContainer" containerID="55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7" Aug 04 20:14:50.362020 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.362000 2580 generic.go:356] "Generic (PLEG): container finished" podID="42ed2bee-87be-44c4-87c6-351d3a71a23b" containerID="00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a" exitCode=0 Aug 04 20:14:50.362089 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.362052 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" Aug 04 20:14:50.362137 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.362055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" event={"ID":"42ed2bee-87be-44c4-87c6-351d3a71a23b","Type":"ContainerDied","Data":"00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a"} Aug 04 20:14:50.362179 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.362158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr" event={"ID":"42ed2bee-87be-44c4-87c6-351d3a71a23b","Type":"ContainerDied","Data":"bd34f6a53b81b8bafdbd8c6e4990cad075bd8ed7fabda1e76f113f858239aabb"} Aug 04 20:14:50.369897 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.369871 2580 scope.go:117] "RemoveContainer" containerID="55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7" Aug 04 20:14:50.370217 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:14:50.370177 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7\": container with ID starting with 55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7 not found: ID does not exist" containerID="55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7" Aug 04 20:14:50.370310 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.370214 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7"} err="failed to get container status \"55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7\": rpc error: code = NotFound desc = could not find container \"55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7\": container with ID starting with 55776482ab74c4424cf4e3be4b50ca0773f8fdd0a9b9d6d4bd5b5016afe83ca7 not found: ID does not exist" Aug 04 20:14:50.370310 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.370240 2580 scope.go:117] "RemoveContainer" containerID="00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a" Aug 04 20:14:50.378369 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.378354 2580 scope.go:117] "RemoveContainer" containerID="00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a" Aug 04 20:14:50.378640 ip-10-0-136-99 kubenswrapper[2580]: E0804 20:14:50.378615 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a\": container with ID starting with 00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a not found: ID does not exist" containerID="00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a" Aug 04 20:14:50.378719 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.378648 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a"} err="failed to get container status \"00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a\": rpc error: code = NotFound desc = could not find container \"00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a\": container with ID starting with 00c565a832c1602a9d2f37b7341647195bc79c206573ee44bcdcc77931e3960a not found: ID does not exist" Aug 04 20:14:50.385986 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.385964 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr"] Aug 04 20:14:50.391275 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.391257 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-15a874-c6858d748-vcrsr"] Aug 04 20:14:50.402999 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.402976 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8"] Aug 04 20:14:50.408895 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:50.408872 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-15a874-5b65dc8c86-vhsd8"] Aug 04 20:14:51.084836 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:51.084806 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:51.603025 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:51.602991 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42ed2bee-87be-44c4-87c6-351d3a71a23b" path="/var/lib/kubelet/pods/42ed2bee-87be-44c4-87c6-351d3a71a23b/volumes" Aug 04 20:14:51.603318 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:51.603305 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="621d2f19-a164-4f12-bcd1-f8c9ff38b76d" path="/var/lib/kubelet/pods/621d2f19-a164-4f12-bcd1-f8c9ff38b76d/volumes" Aug 04 20:14:51.603638 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:51.603625 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="790e1565-3bae-43c2-837f-2364a0a0c875" path="/var/lib/kubelet/pods/790e1565-3bae-43c2-837f-2364a0a0c875/volumes" Aug 04 20:14:55.922650 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:55.922595 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:55.982148 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:55.982123 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:56.209221 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:56.209188 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:56.402128 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:56.402098 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:14:56.800693 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:14:56.800665 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74d77cd6ff-hpnmg_10b57ba5-c8bd-4fa8-b242-a3460419e41d/authorino/0.log" Aug 04 20:15:00.135244 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.135210 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764575-4x8n2"] Aug 04 20:15:00.135689 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.135654 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="621d2f19-a164-4f12-bcd1-f8c9ff38b76d" containerName="revoke-keys" Aug 04 20:15:00.135689 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.135668 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="621d2f19-a164-4f12-bcd1-f8c9ff38b76d" containerName="revoke-keys" Aug 04 20:15:00.135689 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.135679 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ed2bee-87be-44c4-87c6-351d3a71a23b" containerName="payload-pre-processing" Aug 04 20:15:00.135689 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.135685 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ed2bee-87be-44c4-87c6-351d3a71a23b" containerName="payload-pre-processing" Aug 04 20:15:00.135820 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.135693 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790e1565-3bae-43c2-837f-2364a0a0c875" containerName="payload-processing" Aug 04 20:15:00.135820 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.135699 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="790e1565-3bae-43c2-837f-2364a0a0c875" containerName="payload-processing" Aug 04 20:15:00.135820 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.135759 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ed2bee-87be-44c4-87c6-351d3a71a23b" containerName="payload-pre-processing" Aug 04 20:15:00.135820 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.135768 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="790e1565-3bae-43c2-837f-2364a0a0c875" containerName="payload-processing" Aug 04 20:15:00.135820 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.135775 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="621d2f19-a164-4f12-bcd1-f8c9ff38b76d" containerName="revoke-keys" Aug 04 20:15:00.138952 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.138936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764575-4x8n2" Aug 04 20:15:00.141340 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.141316 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 20:15:00.144407 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.144385 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764575-4x8n2"] Aug 04 20:15:00.151288 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.151269 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764575-4x8n2" Aug 04 20:15:00.288299 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.288277 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764575-4x8n2"] Aug 04 20:15:00.291171 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:15:00.291132 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c42535_7c05_4d36_b750_7d60b3ecc00f.slice/crio-8dba6e37d2726555786732a27b8ddf52fe2c66c3416a61a15fcc597a901a5711 WatchSource:0}: Error finding container 8dba6e37d2726555786732a27b8ddf52fe2c66c3416a61a15fcc597a901a5711: Status 404 returned error can't find the container with id 8dba6e37d2726555786732a27b8ddf52fe2c66c3416a61a15fcc597a901a5711 Aug 04 20:15:00.414289 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:00.414208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764575-4x8n2" event={"ID":"00c42535-7c05-4d36-b750-7d60b3ecc00f","Type":"ContainerStarted","Data":"8dba6e37d2726555786732a27b8ddf52fe2c66c3416a61a15fcc597a901a5711"} Aug 04 20:15:01.420530 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:01.420490 2580 generic.go:356] "Generic (PLEG): container finished" podID="00c42535-7c05-4d36-b750-7d60b3ecc00f" containerID="bee0b9b7ffc07e03c6b217d7689dc10ef03ff6fb481ef97cd9ad2abd4c8c2249" exitCode=0 Aug 04 20:15:01.420895 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:01.420545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764575-4x8n2" event={"ID":"00c42535-7c05-4d36-b750-7d60b3ecc00f","Type":"ContainerDied","Data":"bee0b9b7ffc07e03c6b217d7689dc10ef03ff6fb481ef97cd9ad2abd4c8c2249"} Aug 04 20:15:02.176693 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:02.176658 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bbff758-f9jws_2b9a94fb-e69b-4075-affe-0c5f2690c042/manager/0.log" Aug 04 20:15:02.293102 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:02.293078 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rtd5m_648eac31-d4cb-4714-9ef9-fe3f3687fb58/server/0.log" Aug 04 20:15:02.524934 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:02.524902 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-bmz92_2bf84db8-d140-4fdb-a0f8-2fee781b6530/copy-manifests/0.log" Aug 04 20:15:02.532023 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:02.531997 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-bmz92_2bf84db8-d140-4fdb-a0f8-2fee781b6530/manager/0.log" Aug 04 20:15:02.557480 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:02.557442 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764575-4x8n2" Aug 04 20:15:03.132226 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:03.132195 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67c8968587-5lb29_2c659b52-c5e9-483a-8311-d2ae90c856f7/maas-api/0.log" Aug 04 20:15:03.431582 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:03.431505 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764575-4x8n2" event={"ID":"00c42535-7c05-4d36-b750-7d60b3ecc00f","Type":"ContainerDied","Data":"8dba6e37d2726555786732a27b8ddf52fe2c66c3416a61a15fcc597a901a5711"} Aug 04 20:15:03.431582 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:03.431551 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dba6e37d2726555786732a27b8ddf52fe2c66c3416a61a15fcc597a901a5711" Aug 04 20:15:03.431582 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:03.431515 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764575-4x8n2" Aug 04 20:15:03.566004 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:03.565969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764575-4x8n2_00c42535-7c05-4d36-b750-7d60b3ecc00f/cleanup/0.log" Aug 04 20:15:03.674020 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:03.673993 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-f28jh_2f301fa9-6a66-47fb-a7f8-41f8ec6014f2/postgres/0.log" Aug 04 20:15:04.413582 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:04.413551 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74d77cd6ff-hpnmg_10b57ba5-c8bd-4fa8-b242-a3460419e41d/authorino/0.log" Aug 04 20:15:04.749039 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:04.749017 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-8jp4t_0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71/kuadrant-console-plugin/0.log" Aug 04 20:15:05.771908 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:05.771878 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcb8f55f4-rnxjn_6c49c0c7-58e9-491c-9065-44930435bab9/kube-auth-proxy/0.log" Aug 04 20:15:06.001898 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:06.001863 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-xvj9x_f8b01960-83e0-4f88-8923-f300677c5aaf/payload-pre-processing/0.log" Aug 04 20:15:06.118822 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:06.118742 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lqqg_5988f76f-b53e-4784-8e0e-3e1599df97e6/payload-processing/0.log" Aug 04 20:15:06.562403 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:06.562377 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-slt9r_c1557d34-d920-4ee4-8c25-e4fc135bff4f/main/0.log" Aug 04 20:15:06.782349 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:06.782327 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-lbt2k_93540c46-e980-4c34-8704-8cacc2656efc/main/0.log" Aug 04 20:15:07.123082 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:07.123053 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-jc7nl_c7041599-39eb-4372-be28-b4b6032754ab/main/0.log" Aug 04 20:15:13.001805 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:13.001771 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:15:13.104648 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:13.104612 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:15:15.816851 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:15.816817 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r9jp7_241e4560-d288-44ae-af3a-49be302f4d6a/global-pull-secret-syncer/0.log" Aug 04 20:15:15.869004 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:15.868980 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" Aug 04 20:15:15.984511 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:15.984488 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-99.ec2.internal_2f3b2eaee104600060fbf45b71994dca/haproxy/0.log" Aug 04 20:15:20.919862 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:20.919830 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74d77cd6ff-hpnmg_10b57ba5-c8bd-4fa8-b242-a3460419e41d/authorino/0.log" Aug 04 20:15:21.035082 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:21.035056 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-8jp4t_0ce3c2c1-5ba6-4b5f-9c2e-a83659f9da71/kuadrant-console-plugin/0.log" Aug 04 20:15:23.084620 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.084549 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-69xrj_b903dca5-597c-495b-afc5-8f7b573f9ecb/monitoring-plugin/0.log" Aug 04 20:15:23.125392 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.125371 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlpxg_759c21dc-d5ce-48ed-b6be-2acbee04895a/node-exporter/0.log" Aug 04 20:15:23.175585 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.175563 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlpxg_759c21dc-d5ce-48ed-b6be-2acbee04895a/kube-rbac-proxy/0.log" Aug 04 20:15:23.221746 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.221725 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlpxg_759c21dc-d5ce-48ed-b6be-2acbee04895a/init-textfile/0.log" Aug 04 20:15:23.478258 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.478237 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-7nqvm_6edbc642-c8d7-497d-b2a9-fe6c03d34c09/kube-rbac-proxy-main/0.log" Aug 04 20:15:23.503284 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.503262 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-7nqvm_6edbc642-c8d7-497d-b2a9-fe6c03d34c09/kube-rbac-proxy-self/0.log" Aug 04 20:15:23.524846 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.524824 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-7nqvm_6edbc642-c8d7-497d-b2a9-fe6c03d34c09/openshift-state-metrics/0.log" Aug 04 20:15:23.562535 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.562518 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3608a9f-2b8b-4896-a64d-8b3eb9d38b04/prometheus/0.log" Aug 04 20:15:23.583333 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.583316 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3608a9f-2b8b-4896-a64d-8b3eb9d38b04/config-reloader/0.log" Aug 04 20:15:23.604962 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.604937 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3608a9f-2b8b-4896-a64d-8b3eb9d38b04/thanos-sidecar/0.log" Aug 04 20:15:23.624906 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.624888 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3608a9f-2b8b-4896-a64d-8b3eb9d38b04/kube-rbac-proxy-web/0.log" Aug 04 20:15:23.647390 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.647369 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3608a9f-2b8b-4896-a64d-8b3eb9d38b04/kube-rbac-proxy/0.log" Aug 04 20:15:23.675603 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.675585 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3608a9f-2b8b-4896-a64d-8b3eb9d38b04/kube-rbac-proxy-thanos/0.log" Aug 04 20:15:23.698986 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.698965 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3608a9f-2b8b-4896-a64d-8b3eb9d38b04/init-config-reloader/0.log" Aug 04 20:15:23.731747 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.731685 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-5xjhq_51521b28-c77b-4315-a048-d627eafc3751/prometheus-operator/0.log" Aug 04 20:15:23.756068 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.756046 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-5xjhq_51521b28-c77b-4315-a048-d627eafc3751/kube-rbac-proxy/0.log" Aug 04 20:15:23.791075 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.791055 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-gdfl6_a72f3fab-ef8e-44e0-987a-0995ab9b03de/prometheus-operator-admission-webhook/0.log" Aug 04 20:15:23.895704 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.895685 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b49478744-q9h8w_cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a/thanos-query/0.log" Aug 04 20:15:23.918574 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.918557 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b49478744-q9h8w_cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a/kube-rbac-proxy-web/0.log" Aug 04 20:15:23.939121 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.939103 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b49478744-q9h8w_cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a/kube-rbac-proxy/0.log" Aug 04 20:15:23.959456 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.959431 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b49478744-q9h8w_cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a/prom-label-proxy/0.log" Aug 04 20:15:23.984885 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:23.984831 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b49478744-q9h8w_cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a/kube-rbac-proxy-rules/0.log" Aug 04 20:15:24.006126 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:24.006107 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b49478744-q9h8w_cdda85ef-d2f4-4fc2-b6ba-d4b1d00cdc4a/kube-rbac-proxy-metrics/0.log" Aug 04 20:15:25.609127 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.609093 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x"] Aug 04 20:15:25.609508 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.609426 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c42535-7c05-4d36-b750-7d60b3ecc00f" containerName="cleanup" Aug 04 20:15:25.609508 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.609437 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c42535-7c05-4d36-b750-7d60b3ecc00f" containerName="cleanup" Aug 04 20:15:25.609587 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.609515 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c42535-7c05-4d36-b750-7d60b3ecc00f" containerName="cleanup" Aug 04 20:15:25.612788 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.612768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.615105 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.615084 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bk9n8\"/\"kube-root-ca.crt\"" Aug 04 20:15:25.616144 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.616127 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bk9n8\"/\"openshift-service-ca.crt\"" Aug 04 20:15:25.616228 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.616128 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bk9n8\"/\"default-dockercfg-qgnpn\"" Aug 04 20:15:25.621747 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.621727 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x"] Aug 04 20:15:25.710293 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.710257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-proc\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.710496 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.710305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-sys\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.710496 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.710324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-podres\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.710496 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.710343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xz8g\" (UniqueName: \"kubernetes.io/projected/eda82a35-d200-4f1d-9882-6317cb4632a5-kube-api-access-8xz8g\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.710496 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.710411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-lib-modules\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.811403 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.811369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-lib-modules\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.811569 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.811426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-proc\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.811569 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.811458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-sys\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.811569 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.811495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-podres\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.811569 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.811514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xz8g\" (UniqueName: \"kubernetes.io/projected/eda82a35-d200-4f1d-9882-6317cb4632a5-kube-api-access-8xz8g\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.811712 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.811576 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-sys\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.811712 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.811577 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-lib-modules\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.811712 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.811609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-proc\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.811712 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.811628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eda82a35-d200-4f1d-9882-6317cb4632a5-podres\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.819693 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.819662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xz8g\" (UniqueName: \"kubernetes.io/projected/eda82a35-d200-4f1d-9882-6317cb4632a5-kube-api-access-8xz8g\") pod \"perf-node-gather-daemonset-sqs9x\" (UID: \"eda82a35-d200-4f1d-9882-6317cb4632a5\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:25.922622 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:25.922542 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:26.052907 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:26.052884 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x"] Aug 04 20:15:26.055614 ip-10-0-136-99 kubenswrapper[2580]: W0804 20:15:26.055586 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeda82a35_d200_4f1d_9882_6317cb4632a5.slice/crio-dadd12883b11f19d00d9d0b71e9138a372d3b830023ddaf2edb357ad3ae8130f WatchSource:0}: Error finding container dadd12883b11f19d00d9d0b71e9138a372d3b830023ddaf2edb357ad3ae8130f: Status 404 returned error can't find the container with id dadd12883b11f19d00d9d0b71e9138a372d3b830023ddaf2edb357ad3ae8130f Aug 04 20:15:26.541438 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:26.541404 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" event={"ID":"eda82a35-d200-4f1d-9882-6317cb4632a5","Type":"ContainerStarted","Data":"7c6cf671ce185f6fedbb84b6b03251f1bf9d29244cf4dc994caf1b6348601c29"} Aug 04 20:15:26.541662 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:26.541459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" event={"ID":"eda82a35-d200-4f1d-9882-6317cb4632a5","Type":"ContainerStarted","Data":"dadd12883b11f19d00d9d0b71e9138a372d3b830023ddaf2edb357ad3ae8130f"} Aug 04 20:15:26.541662 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:26.541531 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:26.556652 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:26.556611 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" podStartSLOduration=1.556597601 podStartE2EDuration="1.556597601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:15:26.555986292 +0000 UTC m=+3571.461586832" watchObservedRunningTime="2026-08-04 20:15:26.556597601 +0000 UTC m=+3571.462198158" Aug 04 20:15:27.537509 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:27.537459 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:15:27.593755 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:27.593725 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44vkh_29249092-ebd1-4152-b8de-a36c22b99635/konnectivity-agent/0.log" limit=1024 Aug 04 20:15:28.098262 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:28.098225 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sjsm5_456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b/dns/0.log" Aug 04 20:15:28.119667 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:28.119639 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sjsm5_456d2eb0-2a9d-4ff4-a5d5-caef8ddaf85b/kube-rbac-proxy/0.log" Aug 04 20:15:28.170448 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:28.170422 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gk8cr_c261189f-7399-4ff4-a82c-06f9f682305d/dns-node-resolver/0.log" Aug 04 20:15:28.751888 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:28.751861 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvzq8_14c3bacb-2797-4096-ace0-d6e5b61e26be/node-ca/0.log" Aug 04 20:15:30.066751 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:30.066724 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcb8f55f4-rnxjn_6c49c0c7-58e9-491c-9065-44930435bab9/kube-auth-proxy/0.log" Aug 04 20:15:30.142634 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:30.142600 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-xvj9x_f8b01960-83e0-4f88-8923-f300677c5aaf/payload-pre-processing/0.log" Aug 04 20:15:30.176074 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:30.176053 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lqqg_5988f76f-b53e-4784-8e0e-3e1599df97e6/payload-processing/0.log" Aug 04 20:15:31.036476 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:31.036421 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rn9wg_eadc261a-9b4f-493b-a3c9-ba73bcac9dd9/serve-healthcheck-canary/0.log" Aug 04 20:15:31.642889 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:31.642856 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mk4d7_0784792e-39c0-4fa0-8f74-e3a016877e43/kube-rbac-proxy/0.log" Aug 04 20:15:31.661811 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:31.661785 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mk4d7_0784792e-39c0-4fa0-8f74-e3a016877e43/exporter/0.log" Aug 04 20:15:31.682071 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:31.682054 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mk4d7_0784792e-39c0-4fa0-8f74-e3a016877e43/extractor/0.log" Aug 04 20:15:32.556821 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:32.556795 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-sqs9x" Aug 04 20:15:34.606072 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:34.606042 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bbff758-f9jws_2b9a94fb-e69b-4075-affe-0c5f2690c042/manager/0.log" Aug 04 20:15:34.625835 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:34.625774 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rtd5m_648eac31-d4cb-4714-9ef9-fe3f3687fb58/server/0.log" Aug 04 20:15:34.779491 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:34.779434 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-bmz92_2bf84db8-d140-4fdb-a0f8-2fee781b6530/manager/0.log" Aug 04 20:15:34.800599 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:34.800575 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-bmz92_2bf84db8-d140-4fdb-a0f8-2fee781b6530/copy-manifests/0.log" Aug 04 20:15:36.276119 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:36.276077 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59d56c454c-gcq5q_25e3e5d5-e8e1-4bf6-917a-5fdb9a8897c6/manager/0.log" Aug 04 20:15:36.528274 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:36.528187 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcb8f55f4-rnxjn_6c49c0c7-58e9-491c-9065-44930435bab9/kube-auth-proxy/0.log" Aug 04 20:15:36.588902 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:36.588878 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-xvj9x_f8b01960-83e0-4f88-8923-f300677c5aaf/payload-pre-processing/0.log" Aug 04 20:15:36.607015 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:36.606994 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lqqg_5988f76f-b53e-4784-8e0e-3e1599df97e6/payload-processing/0.log" Aug 04 20:15:38.298794 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:38.298767 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcb8f55f4-rnxjn_6c49c0c7-58e9-491c-9065-44930435bab9/kube-auth-proxy/0.log" Aug 04 20:15:38.363305 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:38.363278 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-xvj9x_f8b01960-83e0-4f88-8923-f300677c5aaf/payload-pre-processing/0.log" Aug 04 20:15:38.388927 ip-10-0-136-99 kubenswrapper[2580]: I0804 20:15:38.388907 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lqqg_5988f76f-b53e-4784-8e0e-3e1599df97e6/payload-processing/0.log"