Aug 05 16:20:49.526027 ip-10-0-128-17 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 05 16:20:49.526037 ip-10-0-128-17 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 05 16:20:49.526045 ip-10-0-128-17 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 05 16:20:49.526345 ip-10-0-128-17 systemd[1]: Failed to start Kubernetes Kubelet. Aug 05 16:20:59.645430 ip-10-0-128-17 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 05 16:20:59.645447 ip-10-0-128-17 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c698872e8358487ead089ab5accf57c7 -- Aug 05 16:22:56.528159 ip-10-0-128-17 systemd[1]: Starting Kubernetes Kubelet... Aug 05 16:22:56.901995 ip-10-0-128-17 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 16:22:56.901995 ip-10-0-128-17 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 05 16:22:56.901995 ip-10-0-128-17 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 16:22:56.901995 ip-10-0-128-17 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 05 16:22:56.901995 ip-10-0-128-17 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 16:22:56.904572 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.904457 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 05 16:22:56.909395 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909372 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:56.909395 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909391 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:56.909395 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909395 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:56.909395 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909398 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:56.909395 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909401 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909405 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909408 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909411 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909415 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909417 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909420 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909422 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909425 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909427 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909430 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909432 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909435 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909438 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909440 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909443 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909447 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909450 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909453 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:56.909585 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909456 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909458 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909461 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909463 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909465 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909468 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909471 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909473 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909476 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909479 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909481 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909484 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909486 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909488 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909491 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909494 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909497 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909500 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909502 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909505 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:56.910096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909507 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909510 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909512 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909515 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909517 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909519 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909522 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909525 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909528 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909531 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909533 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909536 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909538 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909541 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909545 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909548 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909551 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909554 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909556 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:56.910573 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909559 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909561 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909564 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909566 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909568 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909571 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909574 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909577 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909581 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909583 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909586 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909590 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909593 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909596 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909599 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909602 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909604 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909607 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909610 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909612 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:56.911068 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909615 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909618 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909621 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909623 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909626 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909628 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909630 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909633 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909636 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909638 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909640 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909643 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909645 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909647 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909650 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909652 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.909654 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910380 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910387 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910390 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:56.911547 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910394 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910396 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910399 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910402 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910405 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910407 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910410 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910412 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910415 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910418 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910420 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910423 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910425 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910428 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910430 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910433 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910435 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910438 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910440 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910442 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:56.912027 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910447 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910451 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910455 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910458 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910461 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910464 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910467 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910469 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910472 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910476 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910479 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910483 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910485 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910489 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910492 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910495 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910498 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910501 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910503 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:56.912500 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910506 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910509 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910512 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910514 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910517 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910520 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910522 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910525 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910528 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910530 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910533 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910535 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910538 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910540 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910542 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910545 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910547 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910550 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910556 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910559 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910562 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:56.912965 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910564 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910566 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910569 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910573 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910576 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910578 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910581 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910584 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910587 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910589 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910592 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910595 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910597 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910599 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910602 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910604 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910606 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910609 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910611 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910614 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:56.913467 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910617 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910619 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910622 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910626 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910628 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910631 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910633 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910638 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910640 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910646 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910648 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910652 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910656 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910659 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910662 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.910664 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911056 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911061 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911065 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:56.914009 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911069 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911072 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911075 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911078 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911081 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911083 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911086 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911089 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911091 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911094 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911096 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911099 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911101 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911104 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911106 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911109 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911111 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911114 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911116 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911120 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:56.914462 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911124 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911126 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911129 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911132 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911134 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911136 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911139 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911141 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911143 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911145 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911150 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911153 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911156 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911158 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911161 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911163 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911165 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911168 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911170 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:56.914995 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911172 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911175 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911177 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911179 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911182 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911184 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911186 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911189 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911192 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911194 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911197 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911199 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911201 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911205 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911207 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911210 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911212 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911215 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911217 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911220 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:56.915428 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911222 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911224 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911227 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911229 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911231 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911234 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911236 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911239 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911241 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911243 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911246 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911248 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911251 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911253 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911255 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911258 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911260 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911262 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911265 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:56.916029 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911268 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911271 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911273 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911275 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911278 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911280 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911283 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911286 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911289 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911292 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911295 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911297 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911300 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911302 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911305 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911307 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911309 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911312 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911376 2575 flags.go:64] FLAG: --address="0.0.0.0" Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911383 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 05 16:22:56.916518 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911388 2575 flags.go:64] FLAG: --anonymous-auth="true" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911392 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911396 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911400 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911403 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911407 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911410 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911413 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911417 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911420 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911423 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911426 2575 flags.go:64] FLAG: --cgroup-root="" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911429 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911432 2575 flags.go:64] FLAG: --client-ca-file="" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911435 2575 flags.go:64] FLAG: --cloud-provider="external" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911438 2575 flags.go:64] FLAG: --cluster-dns="[]" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911443 2575 flags.go:64] FLAG: --cluster-domain="" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911446 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911450 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911453 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911456 2575 flags.go:64] FLAG: --container-log-max-files="5" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911459 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911462 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 05 16:22:56.917003 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911465 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911468 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911471 2575 flags.go:64] FLAG: --contention-profiling="false" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911473 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911476 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911479 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911482 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911485 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911488 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911491 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911494 2575 flags.go:64] FLAG: --enable-load-reader="false" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911496 2575 flags.go:64] FLAG: --enable-server="true" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911499 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911503 2575 flags.go:64] FLAG: --event-burst="100" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911505 2575 flags.go:64] FLAG: --event-qps="50" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911508 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911511 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911514 2575 flags.go:64] FLAG: --eviction-hard="" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911517 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911519 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911522 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911525 2575 flags.go:64] FLAG: --eviction-soft="" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911527 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911530 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911533 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 05 16:22:56.917539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911536 2575 flags.go:64] FLAG: --experimental-mounter-path="" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911538 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911543 2575 flags.go:64] FLAG: --fail-swap-on="true" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911545 2575 flags.go:64] FLAG: --feature-gates="" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911550 2575 flags.go:64] FLAG: --file-check-frequency="20s" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911553 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911556 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911559 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911561 2575 flags.go:64] FLAG: --healthz-port="10248" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911564 2575 flags.go:64] FLAG: --help="false" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911567 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-128-17.ec2.internal" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911570 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911572 2575 flags.go:64] FLAG: --http-check-frequency="20s" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911575 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911578 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911581 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911584 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911586 2575 flags.go:64] FLAG: --image-service-endpoint="" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911589 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911592 2575 flags.go:64] FLAG: --kube-api-burst="100" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911594 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911598 2575 flags.go:64] FLAG: --kube-api-qps="50" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911600 2575 flags.go:64] FLAG: --kube-reserved="" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911603 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 05 16:22:56.918206 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911605 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911608 2575 flags.go:64] FLAG: --kubelet-cgroups="" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911611 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911614 2575 flags.go:64] FLAG: --lock-file="" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911616 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911619 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911622 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911626 2575 flags.go:64] FLAG: --log-json-split-stream="false" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911630 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911632 2575 flags.go:64] FLAG: --log-text-split-stream="false" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911637 2575 flags.go:64] FLAG: --logging-format="text" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911639 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911643 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911646 2575 flags.go:64] FLAG: --manifest-url="" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911649 2575 flags.go:64] FLAG: --manifest-url-header="" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911657 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911660 2575 flags.go:64] FLAG: --max-open-files="1000000" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911664 2575 flags.go:64] FLAG: --max-pods="110" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911666 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911669 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911672 2575 flags.go:64] FLAG: --memory-manager-policy="None" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911675 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911677 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911680 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911683 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 05 16:22:56.918840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911689 2575 flags.go:64] FLAG: --node-status-max-images="50" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911692 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911695 2575 flags.go:64] FLAG: --oom-score-adj="-999" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911698 2575 flags.go:64] FLAG: --pod-cidr="" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911700 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911706 2575 flags.go:64] FLAG: --pod-manifest-path="" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911709 2575 flags.go:64] FLAG: --pod-max-pids="-1" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911712 2575 flags.go:64] FLAG: --pods-per-core="0" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911715 2575 flags.go:64] FLAG: --port="10250" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911717 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911720 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f53e37697cf39a6c" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911723 2575 flags.go:64] FLAG: --qos-reserved="" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911725 2575 flags.go:64] FLAG: --read-only-port="10255" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911728 2575 flags.go:64] FLAG: --register-node="true" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911731 2575 flags.go:64] FLAG: --register-with-taints="" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911734 2575 flags.go:64] FLAG: --registry-burst="10" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911737 2575 flags.go:64] FLAG: --registry-qps="5" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911741 2575 flags.go:64] FLAG: --reserved-cpus="" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911744 2575 flags.go:64] FLAG: --reserved-memory="" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911747 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911751 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911754 2575 flags.go:64] FLAG: --rotate-certificates="false" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911756 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911759 2575 flags.go:64] FLAG: --runonce="false" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911762 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 05 16:22:56.919434 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911764 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911767 2575 flags.go:64] FLAG: --seccomp-default="false" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911770 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911773 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911775 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911778 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911781 2575 flags.go:64] FLAG: --storage-driver-password="root" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911783 2575 flags.go:64] FLAG: --storage-driver-secure="false" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911786 2575 flags.go:64] FLAG: --storage-driver-table="stats" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911802 2575 flags.go:64] FLAG: --storage-driver-user="root" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911805 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911808 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911811 2575 flags.go:64] FLAG: --system-cgroups="" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911813 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911818 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911821 2575 flags.go:64] FLAG: --tls-cert-file="" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911824 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911829 2575 flags.go:64] FLAG: --tls-min-version="" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911831 2575 flags.go:64] FLAG: --tls-private-key-file="" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911834 2575 flags.go:64] FLAG: --topology-manager-policy="none" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911837 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911839 2575 flags.go:64] FLAG: --topology-manager-scope="container" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911842 2575 flags.go:64] FLAG: --v="2" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911846 2575 flags.go:64] FLAG: --version="false" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911851 2575 flags.go:64] FLAG: --vmodule="" Aug 05 16:22:56.920073 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911855 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.911858 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911979 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911983 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911986 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911989 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911992 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911994 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.911997 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912000 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912002 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912005 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912007 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912010 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912013 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912015 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912018 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912020 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912023 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912025 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912027 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:56.920664 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912030 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912032 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912035 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912038 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912040 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912043 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912045 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912047 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912050 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912052 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912056 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912059 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912061 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912065 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912067 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912070 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912072 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912074 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912077 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:56.921168 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912079 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912083 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912086 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912089 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912091 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912094 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912096 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912099 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912101 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912103 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912106 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912108 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912110 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912112 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912115 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912117 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912119 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912122 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912125 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:56.921624 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912127 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912130 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912132 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912135 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912139 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912141 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912144 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912148 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912151 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912153 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912156 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912158 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912160 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912163 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912165 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912167 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912170 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912172 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912175 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912177 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:56.922124 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912179 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912182 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912184 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912188 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912191 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912194 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912196 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912199 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912202 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912204 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912207 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912209 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912211 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912214 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912216 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912219 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912224 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912227 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912230 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:56.922619 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912234 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912238 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.912240 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.912864 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.919615 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.919631 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.919650 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.919655 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919720 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919725 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919728 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919732 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919735 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919738 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919741 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:56.923181 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919743 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919746 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919748 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919751 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919753 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919756 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919760 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919763 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919766 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919769 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919772 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919774 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919777 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919780 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919783 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919786 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919789 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919806 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919809 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:56.923557 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919811 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919814 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919817 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919820 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919823 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919825 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919828 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919830 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919833 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919836 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919838 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919841 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919843 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919846 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919848 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919850 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919853 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919855 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919857 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919860 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:56.924018 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919862 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919864 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919867 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919871 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919874 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919877 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919879 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919882 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919884 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919887 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919890 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919897 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919900 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919902 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919905 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919908 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919911 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919913 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919916 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919918 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:56.924511 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919920 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919922 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919925 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919927 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919929 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919932 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919934 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919936 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919939 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919941 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919943 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919945 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919948 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919950 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919953 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919955 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919957 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919961 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919964 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:56.925096 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919967 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919969 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919972 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919974 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919976 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919983 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919986 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919988 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919991 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919994 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919996 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.919999 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920001 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920003 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.920008 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 16:22:56.925531 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920134 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920139 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920142 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920147 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920149 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920152 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920154 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920157 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920159 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920162 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920164 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920166 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920169 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920171 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920173 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920175 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920178 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920180 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920182 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920185 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:56.925938 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920187 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920190 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920192 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920199 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920202 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920204 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920207 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920209 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920211 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920213 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920217 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920220 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920222 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920225 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920227 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920229 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920231 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920234 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920236 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920239 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:56.926405 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920241 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920243 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920245 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920248 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920250 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920252 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920255 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920258 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920262 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920265 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920267 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920270 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920273 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920276 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920278 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920281 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920288 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920291 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920294 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:56.926930 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920296 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920298 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920300 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920303 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920306 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920308 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920310 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920313 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920315 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920317 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920319 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920321 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920324 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920326 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920328 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920331 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920333 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920336 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920338 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920340 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:56.927388 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920343 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920345 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920347 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920353 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920356 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920358 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920361 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920364 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920372 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920374 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920388 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920391 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920393 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920395 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920398 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920400 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920403 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920405 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920408 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:56.928020 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:56.920410 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:56.928462 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.920415 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 16:22:56.928462 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.920529 2575 server.go:962] "Client rotation is on, will bootstrap in background" Aug 05 16:22:56.928462 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.922547 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 05 16:22:56.928462 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.923948 2575 server.go:1019] "Starting client certificate rotation" Aug 05 16:22:56.928462 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.924055 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 16:22:56.928462 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.924088 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 16:22:56.928462 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.926747 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 16:22:56.932845 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.932820 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 16:22:56.944154 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.944135 2575 log.go:25] "Validated CRI v1 runtime API" Aug 05 16:22:56.946001 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.945983 2575 log.go:25] "Validated CRI v1 image API" Aug 05 16:22:56.946810 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.946779 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 05 16:22:56.949761 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.949740 2575 fs.go:135] Filesystem UUIDs: map[1ad32a30-5f21-4b6d-80d0-6110e423e378:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f68bae0e-86a4-412f-8f23-5a000f1b6615:/dev/nvme0n1p4] Aug 05 16:22:56.949850 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.949759 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 05 16:22:56.952302 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.952283 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 16:22:56.954971 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.954864 2575 manager.go:216] Machine: {Timestamp:2026-08-05 16:22:56.953629003 +0000 UTC m=+0.312880838 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3217054 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21f1962c028e8c69c49147e183ed9e SystemUUID:ec21f196-2c02-8e8c-69c4-9147e183ed9e BootID:c698872e-8358-487e-ad08-9ab5accf57c7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4b:7d:27:2d:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4b:7d:27:2d:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:3a:3f:19:83:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 05 16:22:56.954971 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.954967 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 05 16:22:56.955098 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.955046 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 05 16:22:56.959016 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.958991 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 05 16:22:56.959184 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.959020 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-17.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 05 16:22:56.959866 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.959853 2575 topology_manager.go:138] "Creating topology manager with none policy" Aug 05 16:22:56.959918 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.959871 2575 container_manager_linux.go:309] "Creating device plugin manager" Aug 05 16:22:56.959918 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.959887 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 16:22:56.960622 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.960610 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 16:22:56.961448 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.961436 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 05 16:22:56.964759 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.964745 2575 state_mem.go:36] "Initialized new in-memory state store" Aug 05 16:22:56.964906 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.964895 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 05 16:22:56.964966 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.964932 2575 kubelet.go:486] "Attempting to sync node with API server" Aug 05 16:22:56.964966 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.964947 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 05 16:22:56.964966 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.964961 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 05 16:22:56.965087 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.964974 2575 kubelet.go:398] "Adding apiserver pod source" Aug 05 16:22:56.965087 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.964986 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 05 16:22:56.965357 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.965344 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 16:22:56.965444 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.965368 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 16:22:56.967258 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.967242 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 05 16:22:56.968665 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.968650 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 05 16:22:56.968718 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.968682 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 05 16:22:56.969740 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969723 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 05 16:22:56.969740 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969741 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969746 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969754 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969765 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969773 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969760 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dcdvt" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969780 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969813 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969823 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969828 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969843 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 05 16:22:56.969868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.969849 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 05 16:22:56.970963 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.970953 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 05 16:22:56.970963 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.970962 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 05 16:22:56.974071 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.974056 2575 server.go:1268] "Started kubelet" Aug 05 16:22:56.974166 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.974140 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 05 16:22:56.974232 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.974154 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 05 16:22:56.974232 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.974187 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 05 16:22:56.978188 ip-10-0-128-17 systemd[1]: Started Kubernetes Kubelet. Aug 05 16:22:56.978317 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.978204 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-17.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 05 16:22:56.978387 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:56.978356 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 05 16:22:56.978775 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:56.978699 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 05 16:22:56.979113 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.979069 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 05 16:22:56.979436 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.979413 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dcdvt" Aug 05 16:22:56.981943 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.981926 2575 server.go:310] "Adding debug handlers to kubelet server" Aug 05 16:22:56.984114 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:56.983284 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-17.ec2.internal.18c8f5502e3834fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-17.ec2.internal,UID:ip-10-0-128-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-17.ec2.internal,},FirstTimestamp:2026-08-05 16:22:56.974025982 +0000 UTC m=+0.333277820,LastTimestamp:2026-08-05 16:22:56.974025982 +0000 UTC m=+0.333277820,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-17.ec2.internal,}" Aug 05 16:22:56.985998 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:56.985972 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 05 16:22:56.987255 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.987242 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 05 16:22:56.987301 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.987244 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 05 16:22:56.987358 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:56.987343 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:56.987784 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.987766 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 05 16:22:56.987784 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.987787 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 05 16:22:56.987924 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.987769 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 05 16:22:56.987924 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.987880 2575 reconstruct.go:97] "Volume reconstruction finished" Aug 05 16:22:56.987924 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.987889 2575 reconciler.go:29] "Reconciler: start to sync state" Aug 05 16:22:56.989393 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.989270 2575 factory.go:55] Registering systemd factory Aug 05 16:22:56.989393 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.989283 2575 factory.go:223] Registration of the systemd container factory successfully Aug 05 16:22:56.989658 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.989599 2575 factory.go:153] Registering CRI-O factory Aug 05 16:22:56.989658 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.989613 2575 factory.go:223] Registration of the crio container factory successfully Aug 05 16:22:56.989770 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.989658 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 05 16:22:56.989770 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.989691 2575 factory.go:103] Registering Raw factory Aug 05 16:22:56.989770 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.989724 2575 manager.go:1196] Started watching for new ooms in manager Aug 05 16:22:56.990972 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.990956 2575 manager.go:349] Starting recovery of all containers Aug 05 16:22:56.999351 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:56.999332 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:57.002112 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.001955 2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-17.ec2.internal\" not found" node="ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.002240 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.002226 2575 manager.go:354] Recovery completed Aug 05 16:22:57.006504 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.006489 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:57.008395 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.008379 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:57.008484 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.008413 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:57.008484 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.008436 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:57.009072 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.009057 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 05 16:22:57.009072 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.009070 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 05 16:22:57.009170 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.009086 2575 state_mem.go:36] "Initialized new in-memory state store" Aug 05 16:22:57.009170 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.009090 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:57.010204 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.010181 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:57.010295 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.010221 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:57.010295 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.010238 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:57.010400 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.010383 2575 policy_none.go:49] "None policy: Start" Aug 05 16:22:57.010447 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.010408 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 05 16:22:57.010447 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.010423 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 05 16:22:57.011783 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.011767 2575 policy_none.go:47] "Start" Aug 05 16:22:57.051357 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.051340 2575 manager.go:337] "Starting Device Plugin manager" Aug 05 16:22:57.051479 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.051381 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 05 16:22:57.051479 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.051394 2575 server.go:87] "Starting device plugin registration server" Aug 05 16:22:57.051690 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.051676 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 05 16:22:57.051750 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.051692 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 05 16:22:57.051833 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.051806 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 05 16:22:57.051909 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.051898 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 05 16:22:57.051909 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.051906 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 05 16:22:57.052434 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.052415 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 05 16:22:57.052514 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.052468 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:57.112266 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.112226 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 05 16:22:57.113504 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.113476 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 05 16:22:57.113564 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.113509 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 05 16:22:57.113564 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.113532 2575 kubelet.go:2448] "Starting kubelet main sync loop" Aug 05 16:22:57.113644 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.113568 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 05 16:22:57.116320 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.116298 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:57.152524 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.152452 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:57.153280 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.153264 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:57.153368 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.153300 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:57.153368 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.153316 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:57.153368 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.153346 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.161337 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.161322 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.161400 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.161343 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-17.ec2.internal\": node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:57.190673 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.190646 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:57.214270 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.214243 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal"] Aug 05 16:22:57.214338 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.214322 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:57.215981 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.215968 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:57.216081 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.215995 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:57.216081 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.216003 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:57.217245 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.217232 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:57.217368 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.217354 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.217424 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.217389 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:57.217992 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.217978 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:57.218086 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.217993 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:57.218086 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.218009 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:57.218086 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.218025 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:57.218221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.218012 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:57.218221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.218145 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:57.219960 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.219944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.220005 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.219974 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:57.220590 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.220564 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:57.220686 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.220600 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:57.220686 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.220613 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:57.242317 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.242294 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-17.ec2.internal\" not found" node="ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.246618 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.246604 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-17.ec2.internal\" not found" node="ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.288763 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.288738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c09f632a1212370d17b2b8eceb5b77a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal\" (UID: \"c09f632a1212370d17b2b8eceb5b77a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.288884 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.288769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c09f632a1212370d17b2b8eceb5b77a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal\" (UID: \"c09f632a1212370d17b2b8eceb5b77a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.288884 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.288812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b405ac74232cf257989af7a5ac3a33f2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-17.ec2.internal\" (UID: \"b405ac74232cf257989af7a5ac3a33f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.291752 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.291738 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:57.389353 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.389313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c09f632a1212370d17b2b8eceb5b77a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal\" (UID: \"c09f632a1212370d17b2b8eceb5b77a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.389353 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.389361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c09f632a1212370d17b2b8eceb5b77a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal\" (UID: \"c09f632a1212370d17b2b8eceb5b77a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.389480 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.389378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b405ac74232cf257989af7a5ac3a33f2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-17.ec2.internal\" (UID: \"b405ac74232cf257989af7a5ac3a33f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.389480 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.389417 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b405ac74232cf257989af7a5ac3a33f2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-17.ec2.internal\" (UID: \"b405ac74232cf257989af7a5ac3a33f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.389480 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.389420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c09f632a1212370d17b2b8eceb5b77a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal\" (UID: \"c09f632a1212370d17b2b8eceb5b77a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.389480 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.389420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c09f632a1212370d17b2b8eceb5b77a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal\" (UID: \"c09f632a1212370d17b2b8eceb5b77a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.392415 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.392401 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:57.493257 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.493171 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:57.545367 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.545340 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.550516 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.550490 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal" Aug 05 16:22:57.593589 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.593559 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:57.694131 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.694101 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:57.794671 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.794584 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:57.895225 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.895196 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:57.923380 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.923349 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 05 16:22:57.923863 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.923498 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 16:22:57.923863 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.923520 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 16:22:57.983585 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.983537 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-04 16:17:57 +0000 UTC" deadline="2028-04-16 15:19:00.695660732 +0000 UTC" Aug 05 16:22:57.983585 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:57.983578 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14878h56m2.712085786s" Aug 05 16:22:57.995366 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:57.995346 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-17.ec2.internal\" not found" Aug 05 16:22:58.009619 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.009593 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:58.087888 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.087844 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" Aug 05 16:22:58.100752 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.100730 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 16:22:58.102193 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.102178 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal" Aug 05 16:22:58.106775 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.106751 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:58.110052 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.110032 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 16:22:58.140631 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:58.140601 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb405ac74232cf257989af7a5ac3a33f2.slice/crio-6342910df3cff0aa419b59281c88177ea99514c6a480a44ed00329844279ae4e WatchSource:0}: Error finding container 6342910df3cff0aa419b59281c88177ea99514c6a480a44ed00329844279ae4e: Status 404 returned error can't find the container with id 6342910df3cff0aa419b59281c88177ea99514c6a480a44ed00329844279ae4e Aug 05 16:22:58.145424 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.145410 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:22:58.413567 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:58.413538 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09f632a1212370d17b2b8eceb5b77a7.slice/crio-bc3f730513833f5d402929e4af28044197675caf352576a4e1dd75f62a4a33e0 WatchSource:0}: Error finding container bc3f730513833f5d402929e4af28044197675caf352576a4e1dd75f62a4a33e0: Status 404 returned error can't find the container with id bc3f730513833f5d402929e4af28044197675caf352576a4e1dd75f62a4a33e0 Aug 05 16:22:58.968604 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.968569 2575 apiserver.go:52] "Watching apiserver" Aug 05 16:22:58.978012 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.977985 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 05 16:22:58.978399 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.978376 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jff5g","kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal","openshift-cluster-node-tuning-operator/tuned-trsfg","openshift-image-registry/node-ca-stg9n","openshift-network-diagnostics/network-check-target-57h6j","openshift-network-operator/iptables-alerter-s6ghz","openshift-ovn-kubernetes/ovnkube-node-w7c4d","kube-system/global-pull-secret-syncer-c5l2j","kube-system/konnectivity-agent-559j8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf","openshift-dns/node-resolver-22g2w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal","openshift-multus/multus-additional-cni-plugins-t2jzm","openshift-multus/multus-r4nwq"] Aug 05 16:22:58.979933 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.979867 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:58.980996 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.980964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r4nwq" Aug 05 16:22:58.983048 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.982585 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:58.983048 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.983002 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:22:58.983048 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.983018 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4j4jf\"" Aug 05 16:22:58.983332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.983309 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 05 16:22:58.983947 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.983925 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:58.984849 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.984823 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 05 16:22:58.985364 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.985130 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 05 16:22:58.985364 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.985189 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 05 16:22:58.985364 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.985204 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kcv72\"" Aug 05 16:22:58.985364 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.985272 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 05 16:22:58.985597 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.985378 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x92ct\"" Aug 05 16:22:58.985597 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.985376 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 05 16:22:58.985597 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.985478 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 05 16:22:58.985597 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.985487 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:22:58.985597 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:58.985544 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:22:58.985597 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.985560 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 05 16:22:58.985888 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.985809 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 05 16:22:58.986430 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.986409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:58.986764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.986745 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 05 16:22:58.987040 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.987017 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 05 16:22:58.987997 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.987278 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 05 16:22:58.987997 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.987424 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 05 16:22:58.987997 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.987517 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pvj9p\"" Aug 05 16:22:58.987997 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.987640 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 05 16:22:58.987997 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.987766 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 05 16:22:58.989032 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.988983 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:22:58.989128 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:58.989062 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:22:58.989128 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.989108 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:22:58.989222 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.989148 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 05 16:22:58.989482 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.989462 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 05 16:22:58.989596 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.989512 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sxx66\"" Aug 05 16:22:58.990442 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.990422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:22:58.990605 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:58.990499 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:22:58.992046 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.992029 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-559j8" Aug 05 16:22:58.993904 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.993885 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:58.994773 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.994756 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 05 16:22:58.994976 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.994954 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vp2kf\"" Aug 05 16:22:58.995148 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.995119 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 05 16:22:58.996222 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.996200 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:58.997423 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.997397 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 05 16:22:58.997848 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.997830 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n5jnv\"" Aug 05 16:22:58.997978 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.997958 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 05 16:22:58.998111 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.998091 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.998501 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.998896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-var-lib-kubelet\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.998935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6klsd\" (UniqueName: \"kubernetes.io/projected/637206a4-5a43-4cc0-81eb-e874eef8d544-kube-api-access-6klsd\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.998965 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9768\" (UniqueName: \"kubernetes.io/projected/4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3-kube-api-access-f9768\") pod \"node-ca-stg9n\" (UID: \"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3\") " pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-var-lib-cni-bin\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-var-lib-openvswitch\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999199 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-kubernetes\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-host\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999258 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5wqvc\"" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-slash\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:58.999332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999312 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-cni-netd\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:58.999793 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-var-lib-cni-multus\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:58.999793 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-sysctl-conf\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:58.999793 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-ovnkube-script-lib\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:58.999793 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999611 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7svs\" (UniqueName: \"kubernetes.io/projected/07d7f8ec-bea0-4e1d-9557-325841f4ceaf-kube-api-access-k7svs\") pod \"iptables-alerter-s6ghz\" (UID: \"07d7f8ec-bea0-4e1d-9557-325841f4ceaf\") " pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:58.999793 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3261d122-ae74-432f-89c4-d51c2c4e69f0-tmp\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:58.999793 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-node-log\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:58.999793 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999718 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 05 16:22:58.999793 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-hostroot\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.000093 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-run-netns\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.000093 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:58.999970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-os-release\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.000093 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-daemon-config\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.000093 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000010 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 05 16:22:59.000093 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000053 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x5vc\" (UniqueName: \"kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc\") pod \"network-check-target-57h6j\" (UID: \"25e06ceb-1b89-4317-8789-90457b17cb83\") " pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:22:59.000241 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-sysconfig\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.000241 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-run-ovn\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.000241 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-log-socket\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.000241 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-cni-dir\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.000360 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000246 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-tuned\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.000545 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-run-multus-certs\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.000648 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-etc-kubernetes\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.000648 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-run-openvswitch\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.000763 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-run-k8s-cni-cncf-io\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.000763 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000713 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:22:59.000861 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000766 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxddx\" (UniqueName: \"kubernetes.io/projected/b8676cf2-b5e7-47f2-895e-f5271300ea80-kube-api-access-wxddx\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:22:59.000861 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-env-overrides\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.000978 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/637206a4-5a43-4cc0-81eb-e874eef8d544-cni-binary-copy\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.000978 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-conf-dir\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.000978 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-sysctl-d\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.000978 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.000952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-sys\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.001158 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-systemd-units\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.001158 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-lib-modules\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.001158 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001100 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-etc-openvswitch\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.001158 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-cni-bin\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.001389 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-ovnkube-config\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.001389 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-run-netns\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.001389 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001240 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3-serviceca\") pod \"node-ca-stg9n\" (UID: \"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3\") " pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:59.001389 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001287 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk6vk\" (UniqueName: \"kubernetes.io/projected/3261d122-ae74-432f-89c4-d51c2c4e69f0-kube-api-access-jk6vk\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.001389 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-kubelet\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.001389 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3-host\") pod \"node-ca-stg9n\" (UID: \"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3\") " pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:59.001389 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-run-systemd\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.001698 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.001698 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-system-cni-dir\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.001698 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-cnibin\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.001698 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-socket-dir-parent\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.001698 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-var-lib-kubelet\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.001698 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001660 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bd2tf\"" Aug 05 16:22:59.001698 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-modprobe-d\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.002054 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.002054 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-ovn-node-metrics-cert\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.002054 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07d7f8ec-bea0-4e1d-9557-325841f4ceaf-host-slash\") pod \"iptables-alerter-s6ghz\" (UID: \"07d7f8ec-bea0-4e1d-9557-325841f4ceaf\") " pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:59.002054 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-run\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.002054 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fglz\" (UniqueName: \"kubernetes.io/projected/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-kube-api-access-8fglz\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.002054 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07d7f8ec-bea0-4e1d-9557-325841f4ceaf-iptables-alerter-script\") pod \"iptables-alerter-s6ghz\" (UID: \"07d7f8ec-bea0-4e1d-9557-325841f4ceaf\") " pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:59.002054 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.001979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-systemd\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.002325 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.002112 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 05 16:22:59.002325 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.002130 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 05 16:22:59.005466 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.005441 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 16:22:59.024340 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.024299 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5zs7t" Aug 05 16:22:59.037252 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.037229 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5zs7t" Aug 05 16:22:59.088837 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.088808 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 05 16:22:59.102756 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-systemd\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.102916 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-var-lib-kubelet\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.102916 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6klsd\" (UniqueName: \"kubernetes.io/projected/637206a4-5a43-4cc0-81eb-e874eef8d544-kube-api-access-6klsd\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.102916 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102842 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9768\" (UniqueName: \"kubernetes.io/projected/4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3-kube-api-access-f9768\") pod \"node-ca-stg9n\" (UID: \"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3\") " pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:59.102916 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x27l\" (UniqueName: \"kubernetes.io/projected/9edbc7ed-36b5-4170-8f45-d267b70eac99-kube-api-access-9x27l\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.102916 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102880 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-systemd\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.102916 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-var-lib-cni-bin\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102957 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-var-lib-openvswitch\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-kubernetes\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103014 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-host\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-slash\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-cni-netd\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hm5m\" (UniqueName: \"kubernetes.io/projected/d3c6ce5d-ec80-437e-a440-8cc72d6de221-kube-api-access-7hm5m\") pod \"node-resolver-22g2w\" (UID: \"d3c6ce5d-ec80-437e-a440-8cc72d6de221\") " pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-kubelet-config\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-var-lib-cni-multus\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103168 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-cnibin\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-sysctl-conf\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-ovnkube-script-lib\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.103271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7svs\" (UniqueName: \"kubernetes.io/projected/07d7f8ec-bea0-4e1d-9557-325841f4ceaf-kube-api-access-k7svs\") pod \"iptables-alerter-s6ghz\" (UID: \"07d7f8ec-bea0-4e1d-9557-325841f4ceaf\") " pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3261d122-ae74-432f-89c4-d51c2c4e69f0-tmp\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-node-log\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.102884 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-var-lib-kubelet\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-hostroot\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkv6s\" (UniqueName: \"kubernetes.io/projected/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-kube-api-access-rkv6s\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103417 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-var-lib-openvswitch\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-run-netns\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-os-release\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-daemon-config\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103469 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-run-netns\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5vc\" (UniqueName: \"kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc\") pod \"network-check-target-57h6j\" (UID: \"25e06ceb-1b89-4317-8789-90457b17cb83\") " pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-registration-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-device-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-sysconfig\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-run-ovn\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-log-socket\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.103961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-cni-dir\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103634 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-cni-netd\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-dbus\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103703 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-kubernetes\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-tuned\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-run-multus-certs\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-host\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-etc-kubernetes\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-run-openvswitch\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-run-k8s-cni-cncf-io\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxddx\" (UniqueName: \"kubernetes.io/projected/b8676cf2-b5e7-47f2-895e-f5271300ea80-kube-api-access-wxddx\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103834 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103934 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-var-lib-cni-multus\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104047 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-sysctl-conf\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104309 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-daemon-config\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b60b5da1-b029-4644-b783-f7a393454913-konnectivity-ca\") pod \"konnectivity-agent-559j8\" (UID: \"b60b5da1-b029-4644-b783-f7a393454913\") " pod="kube-system/konnectivity-agent-559j8" Aug 05 16:22:59.104816 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103320 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-slash\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104529 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103394 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-var-lib-cni-bin\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104598 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-ovnkube-script-lib\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104607 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-node-log\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.103776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-os-release\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-hostroot\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-run-k8s-cni-cncf-io\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-env-overrides\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/637206a4-5a43-4cc0-81eb-e874eef8d544-cni-binary-copy\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.104755 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-conf-dir\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-conf-dir\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.104845 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs podName:b8676cf2-b5e7-47f2-895e-f5271300ea80 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:59.604806739 +0000 UTC m=+2.964058585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs") pod "network-metrics-daemon-jff5g" (UID: "b8676cf2-b5e7-47f2-895e-f5271300ea80") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-sysconfig\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.105764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.104943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-run-ovn\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105022 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-etc-kubernetes\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105087 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-log-socket\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105126 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-run-openvswitch\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105278 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-cni-dir\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105330 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-env-overrides\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105348 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-run-multus-certs\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-sysctl-d\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105506 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-sys\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-systemd-units\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-lib-modules\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-etc-openvswitch\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-cni-bin\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105623 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-system-cni-dir\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105610 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-sysctl-d\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-cni-binary-copy\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.106344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105674 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-sys\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-ovnkube-config\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-systemd-units\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105781 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-lib-modules\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105801 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-etc-openvswitch\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105821 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-cni-bin\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105838 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/637206a4-5a43-4cc0-81eb-e874eef8d544-cni-binary-copy\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-run-netns\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-run-netns\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3-serviceca\") pod \"node-ca-stg9n\" (UID: \"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3\") " pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.105951 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-socket-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk6vk\" (UniqueName: \"kubernetes.io/projected/3261d122-ae74-432f-89c4-d51c2c4e69f0-kube-api-access-jk6vk\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-kubelet\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3-host\") pod \"node-ca-stg9n\" (UID: \"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3\") " pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106120 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-kubelet\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b60b5da1-b029-4644-b783-f7a393454913-agent-certs\") pod \"konnectivity-agent-559j8\" (UID: \"b60b5da1-b029-4644-b783-f7a393454913\") " pod="kube-system/konnectivity-agent-559j8" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c6ce5d-ec80-437e-a440-8cc72d6de221-tmp-dir\") pod \"node-resolver-22g2w\" (UID: \"d3c6ce5d-ec80-437e-a440-8cc72d6de221\") " pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106183 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-os-release\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.106921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106184 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-ovnkube-config\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-run-systemd\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-system-cni-dir\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106260 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-cnibin\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-socket-dir-parent\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-var-lib-kubelet\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3-serviceca\") pod \"node-ca-stg9n\" (UID: \"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3\") " pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-sys-fs\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-modprobe-d\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106387 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106388 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3-host\") pod \"node-ca-stg9n\" (UID: \"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3\") " pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-ovn-node-metrics-cert\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07d7f8ec-bea0-4e1d-9557-325841f4ceaf-host-slash\") pod \"iptables-alerter-s6ghz\" (UID: \"07d7f8ec-bea0-4e1d-9557-325841f4ceaf\") " pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106419 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07d7f8ec-bea0-4e1d-9557-325841f4ceaf-host-slash\") pod \"iptables-alerter-s6ghz\" (UID: \"07d7f8ec-bea0-4e1d-9557-325841f4ceaf\") " pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106463 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-host-var-lib-kubelet\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.107702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106520 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-system-cni-dir\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106537 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-modprobe-d\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-cnibin\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-run-systemd\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106580 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/637206a4-5a43-4cc0-81eb-e874eef8d544-multus-socket-dir-parent\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d3c6ce5d-ec80-437e-a440-8cc72d6de221-hosts-file\") pod \"node-resolver-22g2w\" (UID: \"d3c6ce5d-ec80-437e-a440-8cc72d6de221\") " pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-run\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106680 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3261d122-ae74-432f-89c4-d51c2c4e69f0-run\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fglz\" (UniqueName: \"kubernetes.io/projected/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-kube-api-access-8fglz\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.106716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07d7f8ec-bea0-4e1d-9557-325841f4ceaf-iptables-alerter-script\") pod \"iptables-alerter-s6ghz\" (UID: \"07d7f8ec-bea0-4e1d-9557-325841f4ceaf\") " pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.107635 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3261d122-ae74-432f-89c4-d51c2c4e69f0-tmp\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.107652 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07d7f8ec-bea0-4e1d-9557-325841f4ceaf-iptables-alerter-script\") pod \"iptables-alerter-s6ghz\" (UID: \"07d7f8ec-bea0-4e1d-9557-325841f4ceaf\") " pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:59.108617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.107659 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3261d122-ae74-432f-89c4-d51c2c4e69f0-etc-tuned\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.109377 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.109094 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-ovn-node-metrics-cert\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.109913 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.109891 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:22:59.109913 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.109915 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:22:59.110113 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.109930 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2x5vc for pod openshift-network-diagnostics/network-check-target-57h6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:59.110113 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.110025 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc podName:25e06ceb-1b89-4317-8789-90457b17cb83 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:59.609995625 +0000 UTC m=+2.969247451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2x5vc" (UniqueName: "kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc") pod "network-check-target-57h6j" (UID: "25e06ceb-1b89-4317-8789-90457b17cb83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:59.112427 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.112401 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9768\" (UniqueName: \"kubernetes.io/projected/4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3-kube-api-access-f9768\") pod \"node-ca-stg9n\" (UID: \"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3\") " pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:59.113260 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.113232 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7svs\" (UniqueName: \"kubernetes.io/projected/07d7f8ec-bea0-4e1d-9557-325841f4ceaf-kube-api-access-k7svs\") pod \"iptables-alerter-s6ghz\" (UID: \"07d7f8ec-bea0-4e1d-9557-325841f4ceaf\") " pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:59.113359 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.113311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6klsd\" (UniqueName: \"kubernetes.io/projected/637206a4-5a43-4cc0-81eb-e874eef8d544-kube-api-access-6klsd\") pod \"multus-r4nwq\" (UID: \"637206a4-5a43-4cc0-81eb-e874eef8d544\") " pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.116334 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.116307 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxddx\" (UniqueName: \"kubernetes.io/projected/b8676cf2-b5e7-47f2-895e-f5271300ea80-kube-api-access-wxddx\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:22:59.117931 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.117719 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk6vk\" (UniqueName: \"kubernetes.io/projected/3261d122-ae74-432f-89c4-d51c2c4e69f0-kube-api-access-jk6vk\") pod \"tuned-trsfg\" (UID: \"3261d122-ae74-432f-89c4-d51c2c4e69f0\") " pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.118121 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.118097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fglz\" (UniqueName: \"kubernetes.io/projected/c4f66f40-5075-4bbd-9f6d-95c6f63f00ec-kube-api-access-8fglz\") pod \"ovnkube-node-w7c4d\" (UID: \"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.121606 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.121048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" event={"ID":"c09f632a1212370d17b2b8eceb5b77a7","Type":"ContainerStarted","Data":"bc3f730513833f5d402929e4af28044197675caf352576a4e1dd75f62a4a33e0"} Aug 05 16:22:59.122675 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.122652 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal" event={"ID":"b405ac74232cf257989af7a5ac3a33f2","Type":"ContainerStarted","Data":"6342910df3cff0aa419b59281c88177ea99514c6a480a44ed00329844279ae4e"} Aug 05 16:22:59.207149 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b60b5da1-b029-4644-b783-f7a393454913-konnectivity-ca\") pod \"konnectivity-agent-559j8\" (UID: \"b60b5da1-b029-4644-b783-f7a393454913\") " pod="kube-system/konnectivity-agent-559j8" Aug 05 16:22:59.207149 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.207394 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.207394 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.207394 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.207394 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-system-cni-dir\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.207394 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-cni-binary-copy\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.207394 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207359 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.207394 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207359 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.207707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-system-cni-dir\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.207707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-socket-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.207707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207500 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b60b5da1-b029-4644-b783-f7a393454913-agent-certs\") pod \"konnectivity-agent-559j8\" (UID: \"b60b5da1-b029-4644-b783-f7a393454913\") " pod="kube-system/konnectivity-agent-559j8" Aug 05 16:22:59.207707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207523 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.207707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c6ce5d-ec80-437e-a440-8cc72d6de221-tmp-dir\") pod \"node-resolver-22g2w\" (UID: \"d3c6ce5d-ec80-437e-a440-8cc72d6de221\") " pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:59.207707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-os-release\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.207707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-sys-fs\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.207707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d3c6ce5d-ec80-437e-a440-8cc72d6de221-hosts-file\") pod \"node-resolver-22g2w\" (UID: \"d3c6ce5d-ec80-437e-a440-8cc72d6de221\") " pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:59.207707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207700 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x27l\" (UniqueName: \"kubernetes.io/projected/9edbc7ed-36b5-4170-8f45-d267b70eac99-kube-api-access-9x27l\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hm5m\" (UniqueName: \"kubernetes.io/projected/d3c6ce5d-ec80-437e-a440-8cc72d6de221-kube-api-access-7hm5m\") pod \"node-resolver-22g2w\" (UID: \"d3c6ce5d-ec80-437e-a440-8cc72d6de221\") " pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207772 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b60b5da1-b029-4644-b783-f7a393454913-konnectivity-ca\") pod \"konnectivity-agent-559j8\" (UID: \"b60b5da1-b029-4644-b783-f7a393454913\") " pod="kube-system/konnectivity-agent-559j8" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-kubelet-config\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207790 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-sys-fs\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-cnibin\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-kubelet-config\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkv6s\" (UniqueName: \"kubernetes.io/projected/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-kube-api-access-rkv6s\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207883 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-os-release\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c6ce5d-ec80-437e-a440-8cc72d6de221-tmp-dir\") pod \"node-resolver-22g2w\" (UID: \"d3c6ce5d-ec80-437e-a440-8cc72d6de221\") " pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207908 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d3c6ce5d-ec80-437e-a440-8cc72d6de221-hosts-file\") pod \"node-resolver-22g2w\" (UID: \"d3c6ce5d-ec80-437e-a440-8cc72d6de221\") " pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-registration-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-device-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.207985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-dbus\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.208013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:22:59.208139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.208057 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-device-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.208809 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.208014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-socket-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.208809 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.208132 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-dbus\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:22:59.208809 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.208134 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9edbc7ed-36b5-4170-8f45-d267b70eac99-registration-dir\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.208809 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.208283 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-cnibin\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.208809 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.208304 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:59.208809 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.208384 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret podName:8ddce42c-0420-4820-9d02-f1bfbdd3bf42 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:59.70836453 +0000 UTC m=+3.067616370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret") pod "global-pull-secret-syncer-c5l2j" (UID: "8ddce42c-0420-4820-9d02-f1bfbdd3bf42") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:59.208809 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.208518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-cni-binary-copy\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.208809 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.208522 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.210952 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.210926 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b60b5da1-b029-4644-b783-f7a393454913-agent-certs\") pod \"konnectivity-agent-559j8\" (UID: \"b60b5da1-b029-4644-b783-f7a393454913\") " pod="kube-system/konnectivity-agent-559j8" Aug 05 16:22:59.217746 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.217698 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x27l\" (UniqueName: \"kubernetes.io/projected/9edbc7ed-36b5-4170-8f45-d267b70eac99-kube-api-access-9x27l\") pod \"aws-ebs-csi-driver-node-m8sqf\" (UID: \"9edbc7ed-36b5-4170-8f45-d267b70eac99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.218000 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.217949 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hm5m\" (UniqueName: \"kubernetes.io/projected/d3c6ce5d-ec80-437e-a440-8cc72d6de221-kube-api-access-7hm5m\") pod \"node-resolver-22g2w\" (UID: \"d3c6ce5d-ec80-437e-a440-8cc72d6de221\") " pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:59.218136 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.218115 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkv6s\" (UniqueName: \"kubernetes.io/projected/6bd12d48-f8bc-4866-ac63-a9b61e696eb0-kube-api-access-rkv6s\") pod \"multus-additional-cni-plugins-t2jzm\" (UID: \"6bd12d48-f8bc-4866-ac63-a9b61e696eb0\") " pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.277660 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.277579 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:59.293307 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.293282 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-trsfg" Aug 05 16:22:59.305943 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.305917 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r4nwq" Aug 05 16:22:59.306228 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:59.306206 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3261d122_ae74_432f_89c4_d51c2c4e69f0.slice/crio-11e996030b72b9c23a4298c4d436952ddc110f235ea8128c061108916cff1a39 WatchSource:0}: Error finding container 11e996030b72b9c23a4298c4d436952ddc110f235ea8128c061108916cff1a39: Status 404 returned error can't find the container with id 11e996030b72b9c23a4298c4d436952ddc110f235ea8128c061108916cff1a39 Aug 05 16:22:59.314069 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.314007 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:22:59.315233 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:59.315195 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637206a4_5a43_4cc0_81eb_e874eef8d544.slice/crio-2b71d01fed7535abb688cd290d9b3065bab3df8fa30fa729d96304e13a1091c1 WatchSource:0}: Error finding container 2b71d01fed7535abb688cd290d9b3065bab3df8fa30fa729d96304e13a1091c1: Status 404 returned error can't find the container with id 2b71d01fed7535abb688cd290d9b3065bab3df8fa30fa729d96304e13a1091c1 Aug 05 16:22:59.321303 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.321283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-stg9n" Aug 05 16:22:59.324105 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:59.324083 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f66f40_5075_4bbd_9f6d_95c6f63f00ec.slice/crio-204bb255eb65a0b6b1d562e51b999d9d819a0d9722d2ccf87335d048b9f6c46b WatchSource:0}: Error finding container 204bb255eb65a0b6b1d562e51b999d9d819a0d9722d2ccf87335d048b9f6c46b: Status 404 returned error can't find the container with id 204bb255eb65a0b6b1d562e51b999d9d819a0d9722d2ccf87335d048b9f6c46b Aug 05 16:22:59.329475 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.329424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s6ghz" Aug 05 16:22:59.331693 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:59.331664 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4873e7f5_62e1_4a7f_99c8_1f67a7fd86e3.slice/crio-a5cd7940f8598ceaa9d596d0c42eedd820c5108891163edca0fc055b8011f8f5 WatchSource:0}: Error finding container a5cd7940f8598ceaa9d596d0c42eedd820c5108891163edca0fc055b8011f8f5: Status 404 returned error can't find the container with id a5cd7940f8598ceaa9d596d0c42eedd820c5108891163edca0fc055b8011f8f5 Aug 05 16:22:59.339219 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:59.339194 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d7f8ec_bea0_4e1d_9557_325841f4ceaf.slice/crio-b690613c1787fafc062d8d9a206b47d4ece66957b7dbcd33364d6b54879181b2 WatchSource:0}: Error finding container b690613c1787fafc062d8d9a206b47d4ece66957b7dbcd33364d6b54879181b2: Status 404 returned error can't find the container with id b690613c1787fafc062d8d9a206b47d4ece66957b7dbcd33364d6b54879181b2 Aug 05 16:22:59.345275 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.345251 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-559j8" Aug 05 16:22:59.353008 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.352986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" Aug 05 16:22:59.354023 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:59.353999 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60b5da1_b029_4644_b783_f7a393454913.slice/crio-428ebd221fa8524cd27b73191a404f125a0d994e323844c7b498eaf76b0b2178 WatchSource:0}: Error finding container 428ebd221fa8524cd27b73191a404f125a0d994e323844c7b498eaf76b0b2178: Status 404 returned error can't find the container with id 428ebd221fa8524cd27b73191a404f125a0d994e323844c7b498eaf76b0b2178 Aug 05 16:22:59.359643 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.359613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-22g2w" Aug 05 16:22:59.361899 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:59.361851 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edbc7ed_36b5_4170_8f45_d267b70eac99.slice/crio-be1ab4620457347c66abaaab7e23808bd3f938a784c846bae9f3c52ba9e10381 WatchSource:0}: Error finding container be1ab4620457347c66abaaab7e23808bd3f938a784c846bae9f3c52ba9e10381: Status 404 returned error can't find the container with id be1ab4620457347c66abaaab7e23808bd3f938a784c846bae9f3c52ba9e10381 Aug 05 16:22:59.366261 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.366240 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" Aug 05 16:22:59.373056 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:59.373032 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c6ce5d_ec80_437e_a440_8cc72d6de221.slice/crio-dc242d0e86269fee186de4dce2b7a556a26da610b9241d41b3ac96d100e412d2 WatchSource:0}: Error finding container dc242d0e86269fee186de4dce2b7a556a26da610b9241d41b3ac96d100e412d2: Status 404 returned error can't find the container with id dc242d0e86269fee186de4dce2b7a556a26da610b9241d41b3ac96d100e412d2 Aug 05 16:22:59.380186 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:22:59.380158 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd12d48_f8bc_4866_ac63_a9b61e696eb0.slice/crio-f538161b6323344ae5a4c1c13406065ee3034d9ed0e8563a9b5d50a89b86a923 WatchSource:0}: Error finding container f538161b6323344ae5a4c1c13406065ee3034d9ed0e8563a9b5d50a89b86a923: Status 404 returned error can't find the container with id f538161b6323344ae5a4c1c13406065ee3034d9ed0e8563a9b5d50a89b86a923 Aug 05 16:22:59.470235 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.470205 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:59.611757 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.611643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5vc\" (UniqueName: \"kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc\") pod \"network-check-target-57h6j\" (UID: \"25e06ceb-1b89-4317-8789-90457b17cb83\") " pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:22:59.611757 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.611717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:22:59.612006 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.611837 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:59.612006 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.611870 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:22:59.612006 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.611896 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:22:59.612006 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.611908 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2x5vc for pod openshift-network-diagnostics/network-check-target-57h6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:59.612006 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.611929 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs podName:b8676cf2-b5e7-47f2-895e-f5271300ea80 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:00.611908701 +0000 UTC m=+3.971160550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs") pod "network-metrics-daemon-jff5g" (UID: "b8676cf2-b5e7-47f2-895e-f5271300ea80") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:59.612006 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.611953 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc podName:25e06ceb-1b89-4317-8789-90457b17cb83 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:00.611941267 +0000 UTC m=+3.971193091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x5vc" (UniqueName: "kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc") pod "network-check-target-57h6j" (UID: "25e06ceb-1b89-4317-8789-90457b17cb83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:59.712039 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:22:59.712006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:22:59.712223 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.712156 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:59.712279 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:22:59.712228 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret podName:8ddce42c-0420-4820-9d02-f1bfbdd3bf42 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:00.712205735 +0000 UTC m=+4.071457559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret") pod "global-pull-secret-syncer-c5l2j" (UID: "8ddce42c-0420-4820-9d02-f1bfbdd3bf42") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:00.039962 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.039916 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 16:17:59 +0000 UTC" deadline="2028-03-30 07:27:12.887053997 +0000 UTC" Aug 05 16:23:00.039962 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.039959 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14463h4m12.847100063s" Aug 05 16:23:00.126859 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.126798 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-trsfg" event={"ID":"3261d122-ae74-432f-89c4-d51c2c4e69f0","Type":"ContainerStarted","Data":"11e996030b72b9c23a4298c4d436952ddc110f235ea8128c061108916cff1a39"} Aug 05 16:23:00.129558 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.129525 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-22g2w" event={"ID":"d3c6ce5d-ec80-437e-a440-8cc72d6de221","Type":"ContainerStarted","Data":"dc242d0e86269fee186de4dce2b7a556a26da610b9241d41b3ac96d100e412d2"} Aug 05 16:23:00.132385 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.132357 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r4nwq" event={"ID":"637206a4-5a43-4cc0-81eb-e874eef8d544","Type":"ContainerStarted","Data":"2b71d01fed7535abb688cd290d9b3065bab3df8fa30fa729d96304e13a1091c1"} Aug 05 16:23:00.134144 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.134116 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" event={"ID":"6bd12d48-f8bc-4866-ac63-a9b61e696eb0","Type":"ContainerStarted","Data":"f538161b6323344ae5a4c1c13406065ee3034d9ed0e8563a9b5d50a89b86a923"} Aug 05 16:23:00.135882 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.135856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" event={"ID":"9edbc7ed-36b5-4170-8f45-d267b70eac99","Type":"ContainerStarted","Data":"be1ab4620457347c66abaaab7e23808bd3f938a784c846bae9f3c52ba9e10381"} Aug 05 16:23:00.137386 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.137362 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-559j8" event={"ID":"b60b5da1-b029-4644-b783-f7a393454913","Type":"ContainerStarted","Data":"428ebd221fa8524cd27b73191a404f125a0d994e323844c7b498eaf76b0b2178"} Aug 05 16:23:00.138862 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.138832 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s6ghz" event={"ID":"07d7f8ec-bea0-4e1d-9557-325841f4ceaf","Type":"ContainerStarted","Data":"b690613c1787fafc062d8d9a206b47d4ece66957b7dbcd33364d6b54879181b2"} Aug 05 16:23:00.140377 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.140338 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-stg9n" event={"ID":"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3","Type":"ContainerStarted","Data":"a5cd7940f8598ceaa9d596d0c42eedd820c5108891163edca0fc055b8011f8f5"} Aug 05 16:23:00.142384 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.142323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" event={"ID":"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec","Type":"ContainerStarted","Data":"204bb255eb65a0b6b1d562e51b999d9d819a0d9722d2ccf87335d048b9f6c46b"} Aug 05 16:23:00.618896 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.618854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5vc\" (UniqueName: \"kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc\") pod \"network-check-target-57h6j\" (UID: \"25e06ceb-1b89-4317-8789-90457b17cb83\") " pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:00.619067 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.618935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:00.619138 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:00.619072 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:00.619138 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:00.619136 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs podName:b8676cf2-b5e7-47f2-895e-f5271300ea80 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:02.619117069 +0000 UTC m=+5.978368896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs") pod "network-metrics-daemon-jff5g" (UID: "b8676cf2-b5e7-47f2-895e-f5271300ea80") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:00.619609 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:00.619586 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:23:00.619609 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:00.619612 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:23:00.619763 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:00.619625 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2x5vc for pod openshift-network-diagnostics/network-check-target-57h6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:00.619763 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:00.619674 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc podName:25e06ceb-1b89-4317-8789-90457b17cb83 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:02.619657748 +0000 UTC m=+5.978909578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x5vc" (UniqueName: "kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc") pod "network-check-target-57h6j" (UID: "25e06ceb-1b89-4317-8789-90457b17cb83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:00.719813 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:00.719724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:00.720022 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:00.719991 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:00.720086 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:00.720065 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret podName:8ddce42c-0420-4820-9d02-f1bfbdd3bf42 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:02.720044724 +0000 UTC m=+6.079296553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret") pod "global-pull-secret-syncer-c5l2j" (UID: "8ddce42c-0420-4820-9d02-f1bfbdd3bf42") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:01.040645 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:01.040518 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 16:17:59 +0000 UTC" deadline="2028-04-10 15:44:20.515287562 +0000 UTC" Aug 05 16:23:01.040645 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:01.040606 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14735h21m19.474686662s" Aug 05 16:23:01.114696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:01.114658 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:01.114894 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:01.114791 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:01.115289 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:01.115261 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:01.115451 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:01.115389 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:01.117595 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:01.117569 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:01.117701 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:01.117675 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:02.639240 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:02.639071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:02.639240 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:02.639156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5vc\" (UniqueName: \"kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc\") pod \"network-check-target-57h6j\" (UID: \"25e06ceb-1b89-4317-8789-90457b17cb83\") " pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:02.639823 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:02.639313 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:23:02.639823 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:02.639331 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:23:02.639823 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:02.639345 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2x5vc for pod openshift-network-diagnostics/network-check-target-57h6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:02.639823 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:02.639406 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc podName:25e06ceb-1b89-4317-8789-90457b17cb83 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:06.639386082 +0000 UTC m=+9.998637926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x5vc" (UniqueName: "kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc") pod "network-check-target-57h6j" (UID: "25e06ceb-1b89-4317-8789-90457b17cb83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:02.639823 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:02.639797 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:02.640122 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:02.639844 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs podName:b8676cf2-b5e7-47f2-895e-f5271300ea80 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:06.639827574 +0000 UTC m=+9.999079402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs") pod "network-metrics-daemon-jff5g" (UID: "b8676cf2-b5e7-47f2-895e-f5271300ea80") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:02.740306 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:02.740264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:02.740494 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:02.740469 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:02.740554 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:02.740534 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret podName:8ddce42c-0420-4820-9d02-f1bfbdd3bf42 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:06.740514041 +0000 UTC m=+10.099765878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret") pod "global-pull-secret-syncer-c5l2j" (UID: "8ddce42c-0420-4820-9d02-f1bfbdd3bf42") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:03.115612 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:03.115047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:03.115612 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:03.115191 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:03.115612 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:03.115248 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:03.115612 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:03.115370 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:03.115612 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:03.115420 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:03.115612 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:03.115509 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:05.114041 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:05.113993 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:05.114505 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:05.114140 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:05.114505 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:05.113993 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:05.114642 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:05.114512 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:05.114700 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:05.114661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:05.114755 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:05.114729 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:06.672898 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:06.672284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5vc\" (UniqueName: \"kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc\") pod \"network-check-target-57h6j\" (UID: \"25e06ceb-1b89-4317-8789-90457b17cb83\") " pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:06.672898 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:06.672345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:06.672898 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:06.672494 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:06.672898 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:06.672512 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:23:06.672898 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:06.672539 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:23:06.672898 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:06.672554 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2x5vc for pod openshift-network-diagnostics/network-check-target-57h6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:06.672898 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:06.672558 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs podName:b8676cf2-b5e7-47f2-895e-f5271300ea80 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:14.672536385 +0000 UTC m=+18.031788216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs") pod "network-metrics-daemon-jff5g" (UID: "b8676cf2-b5e7-47f2-895e-f5271300ea80") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:06.672898 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:06.672612 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc podName:25e06ceb-1b89-4317-8789-90457b17cb83 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:14.672592615 +0000 UTC m=+18.031844463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x5vc" (UniqueName: "kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc") pod "network-check-target-57h6j" (UID: "25e06ceb-1b89-4317-8789-90457b17cb83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:06.773390 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:06.773345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:06.773602 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:06.773533 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:06.773648 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:06.773621 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret podName:8ddce42c-0420-4820-9d02-f1bfbdd3bf42 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:14.773600446 +0000 UTC m=+18.132852275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret") pod "global-pull-secret-syncer-c5l2j" (UID: "8ddce42c-0420-4820-9d02-f1bfbdd3bf42") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:07.114985 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:07.114899 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:07.115129 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:07.115020 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:07.115411 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:07.115392 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:07.115537 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:07.115508 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:07.115663 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:07.115559 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:07.115663 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:07.115628 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:09.115167 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:09.115133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:09.115582 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:09.115241 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:09.115582 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:09.115136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:09.115582 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:09.115263 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:09.115582 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:09.115311 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:09.115582 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:09.115358 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:11.114634 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:11.114597 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:11.115102 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:11.114645 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:11.115102 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:11.114599 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:11.115102 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:11.114737 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:11.115102 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:11.114831 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:11.115102 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:11.114920 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:13.114685 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:13.114649 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:13.115111 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:13.114649 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:13.115111 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:13.114751 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:13.115111 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:13.114649 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:13.115111 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:13.114887 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:13.115111 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:13.114978 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:14.734886 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:14.734853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5vc\" (UniqueName: \"kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc\") pod \"network-check-target-57h6j\" (UID: \"25e06ceb-1b89-4317-8789-90457b17cb83\") " pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:14.734886 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:14.734899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:14.735315 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:14.734997 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:14.735315 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:14.735018 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:23:14.735315 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:14.735037 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:23:14.735315 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:14.735047 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2x5vc for pod openshift-network-diagnostics/network-check-target-57h6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:14.735315 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:14.735052 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs podName:b8676cf2-b5e7-47f2-895e-f5271300ea80 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:30.73503631 +0000 UTC m=+34.094288136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs") pod "network-metrics-daemon-jff5g" (UID: "b8676cf2-b5e7-47f2-895e-f5271300ea80") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:14.735315 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:14.735086 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc podName:25e06ceb-1b89-4317-8789-90457b17cb83 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:30.73507361 +0000 UTC m=+34.094325432 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x5vc" (UniqueName: "kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc") pod "network-check-target-57h6j" (UID: "25e06ceb-1b89-4317-8789-90457b17cb83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:14.835438 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:14.835403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:14.835581 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:14.835561 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:14.835644 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:14.835633 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret podName:8ddce42c-0420-4820-9d02-f1bfbdd3bf42 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:30.835614866 +0000 UTC m=+34.194866702 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret") pod "global-pull-secret-syncer-c5l2j" (UID: "8ddce42c-0420-4820-9d02-f1bfbdd3bf42") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:15.114700 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:15.114615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:15.114700 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:15.114644 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:15.114891 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:15.114615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:15.114891 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:15.114741 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:15.114891 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:15.114804 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:15.114891 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:15.114871 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:17.115120 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:17.115082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:17.115513 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:17.115167 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:17.115513 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:17.115182 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:17.115513 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:17.115301 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:17.115513 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:17.115358 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:17.115513 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:17.115426 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:19.118359 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:19.118332 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:19.118703 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:19.118343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:19.118703 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:19.118452 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:19.118703 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:19.118436 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:19.118703 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:19.118537 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:19.118703 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:19.118592 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:19.190730 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:19.190698 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" event={"ID":"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec","Type":"ContainerStarted","Data":"b8023fae7a0fe4d7d08c588b2a036781217ccd7f01827e5ae0c4db573630fb3c"} Aug 05 16:23:19.192136 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:19.192111 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-trsfg" event={"ID":"3261d122-ae74-432f-89c4-d51c2c4e69f0","Type":"ContainerStarted","Data":"d03df06cadd37d398ac8ea48023681c2cf9c98759baf91646d833f9b69e9f186"} Aug 05 16:23:19.193348 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:19.193324 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal" event={"ID":"b405ac74232cf257989af7a5ac3a33f2","Type":"ContainerStarted","Data":"fffef0174fbde10db006594ce7c6da35a6ed5ed9a4fb914a9aa22e9091c7f60a"} Aug 05 16:23:19.210500 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:19.210453 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-trsfg" podStartSLOduration=2.5338063 podStartE2EDuration="22.210424994s" totalImagesPullingTime="19.676618694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:19.210400308 +0000 UTC m=+22.569652152" watchObservedRunningTime="2026-08-05 16:23:19.210424994 +0000 UTC m=+22.569676819" Aug 05 16:23:19.224840 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:19.224755 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-17.ec2.internal" podStartSLOduration=21.224741816 podStartE2EDuration="21.224741816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:19.224302968 +0000 UTC m=+22.583554811" watchObservedRunningTime="2026-08-05 16:23:19.224741816 +0000 UTC m=+22.583993661" Aug 05 16:23:20.196742 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.196548 2575 generic.go:356] "Generic (PLEG): container finished" podID="c09f632a1212370d17b2b8eceb5b77a7" containerID="63b610d0ef8d59e3e5a8b7be79346b9a1b128165c7d4f92407a3b5f8c49e45c5" exitCode=0 Aug 05 16:23:20.197416 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.196621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" event={"ID":"c09f632a1212370d17b2b8eceb5b77a7","Type":"ContainerDied","Data":"63b610d0ef8d59e3e5a8b7be79346b9a1b128165c7d4f92407a3b5f8c49e45c5"} Aug 05 16:23:20.198458 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.198434 2575 generic.go:356] "Generic (PLEG): container finished" podID="6bd12d48-f8bc-4866-ac63-a9b61e696eb0" containerID="0810bd2c92337bba7a4b73f484c9a2c8863aa782d339312d7b1d84798a31d82e" exitCode=0 Aug 05 16:23:20.198567 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.198512 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" event={"ID":"6bd12d48-f8bc-4866-ac63-a9b61e696eb0","Type":"ContainerDied","Data":"0810bd2c92337bba7a4b73f484c9a2c8863aa782d339312d7b1d84798a31d82e"} Aug 05 16:23:20.200273 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.200254 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" event={"ID":"9edbc7ed-36b5-4170-8f45-d267b70eac99","Type":"ContainerStarted","Data":"5e51ab38bbb41439cd52fd3d559580a1571a3f353016882a11294175defaee12"} Aug 05 16:23:20.201644 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.201624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-559j8" event={"ID":"b60b5da1-b029-4644-b783-f7a393454913","Type":"ContainerStarted","Data":"2ccafa546c43d20cd9512afc6424cc7c348d1a826adbe960a11bb0615221bc7c"} Aug 05 16:23:20.204252 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.204225 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-stg9n" event={"ID":"4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3","Type":"ContainerStarted","Data":"35d90a99ea69ba15b1528f61bcc3eda5772c9212fe72c1d611c3779a9b48f77a"} Aug 05 16:23:20.206597 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.206580 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:23:20.206874 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.206858 2575 generic.go:356] "Generic (PLEG): container finished" podID="c4f66f40-5075-4bbd-9f6d-95c6f63f00ec" containerID="d1b62bad08ac9b3ce7065cb00be08130bd0fcfdb74d709d858ac7d315afb41dc" exitCode=1 Aug 05 16:23:20.206952 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.206921 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" event={"ID":"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec","Type":"ContainerStarted","Data":"0f85cc3dde48edb15f389e6f919d1f0185dde17842101aa7c6c2a3855be575d2"} Aug 05 16:23:20.206994 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.206971 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" event={"ID":"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec","Type":"ContainerStarted","Data":"474e48e4af9dce0cad5d701b8180cc46f4dc16b0b976d2faa41036103084a1e7"} Aug 05 16:23:20.207026 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.207004 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" event={"ID":"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec","Type":"ContainerStarted","Data":"e8bf67c3457a81561792f7eda1ad2b78bfeb434ea3665cb7361a921440b5f0bd"} Aug 05 16:23:20.207066 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.207029 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" event={"ID":"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec","Type":"ContainerStarted","Data":"2c49b24861286e1dadc91f840129e9805e711e41824a3a94f7f88cfc2bbe6b5e"} Aug 05 16:23:20.207066 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.207056 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" event={"ID":"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec","Type":"ContainerDied","Data":"d1b62bad08ac9b3ce7065cb00be08130bd0fcfdb74d709d858ac7d315afb41dc"} Aug 05 16:23:20.208216 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.208193 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-22g2w" event={"ID":"d3c6ce5d-ec80-437e-a440-8cc72d6de221","Type":"ContainerStarted","Data":"0d5539c066524bd62bbf93b66f1af2f629c719c706ef625699e4a1c2ed5b2056"} Aug 05 16:23:20.209433 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.209414 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r4nwq" event={"ID":"637206a4-5a43-4cc0-81eb-e874eef8d544","Type":"ContainerStarted","Data":"12f52939f0b3e427c819ba69c6ae9a45c175a66042d439fb6eebf04fb4914e17"} Aug 05 16:23:20.232159 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.231968 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-stg9n" podStartSLOduration=2.583146546 podStartE2EDuration="22.231956237s" totalImagesPullingTime="19.648809691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:20.231729062 +0000 UTC m=+23.590980907" watchObservedRunningTime="2026-08-05 16:23:20.231956237 +0000 UTC m=+23.591208081" Aug 05 16:23:20.270329 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.270277 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-559j8" podStartSLOduration=7.0112005 podStartE2EDuration="22.270259387s" totalImagesPullingTime="15.259058887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:20.269105184 +0000 UTC m=+23.628357029" watchObservedRunningTime="2026-08-05 16:23:20.270259387 +0000 UTC m=+23.629511231" Aug 05 16:23:20.290575 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.290532 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r4nwq" podStartSLOduration=3.235961106 podStartE2EDuration="23.290517177s" totalImagesPullingTime="20.054556071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:20.290177439 +0000 UTC m=+23.649429285" watchObservedRunningTime="2026-08-05 16:23:20.290517177 +0000 UTC m=+23.649769021" Aug 05 16:23:20.307283 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.307231 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-22g2w" podStartSLOduration=2.709237993 podStartE2EDuration="22.307217425s" totalImagesPullingTime="19.597979432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:20.306821591 +0000 UTC m=+23.666073436" watchObservedRunningTime="2026-08-05 16:23:20.307217425 +0000 UTC m=+23.666469268" Aug 05 16:23:20.319004 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.318973 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:20.410355 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.410327 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:20.468559 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:20.468525 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:21.120494 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:21.120472 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:21.120633 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:21.120555 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:21.120633 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:21.120562 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:21.120633 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:21.120578 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:21.120728 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:21.120643 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:21.120728 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:21.120703 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:21.168040 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:21.168016 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 05 16:23:21.213668 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:21.213638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" event={"ID":"9edbc7ed-36b5-4170-8f45-d267b70eac99","Type":"ContainerStarted","Data":"8afa6771447936c649c3d87d6a084465095d5bcf99f8570d80085e54762ee5dc"} Aug 05 16:23:21.215410 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:21.215374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s6ghz" event={"ID":"07d7f8ec-bea0-4e1d-9557-325841f4ceaf","Type":"ContainerStarted","Data":"5ba25b2baf39c5eb57d4baf2c004d79928c267891e193e15339f95219d7499a5"} Aug 05 16:23:21.217245 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:21.217222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" event={"ID":"c09f632a1212370d17b2b8eceb5b77a7","Type":"ContainerStarted","Data":"3aeb9abe041acdc69ce4704f00b5b578846313f3fe6262c81daead81b575ac49"} Aug 05 16:23:21.231137 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:21.231098 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s6ghz" podStartSLOduration=3.60065395 podStartE2EDuration="23.23108609s" totalImagesPullingTime="19.63043214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:21.230839508 +0000 UTC m=+24.590091353" watchObservedRunningTime="2026-08-05 16:23:21.23108609 +0000 UTC m=+24.590337963" Aug 05 16:23:21.350086 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:21.350002 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:21.406912 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:21.406876 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:22.063343 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:22.063198 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-05T16:23:21.168039158Z","UUID":"e3ab7010-a2bb-4803-b416-60f98a5f97fb","Handler":null,"Name":"","Endpoint":""} Aug 05 16:23:22.064899 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:22.064851 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 05 16:23:22.064899 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:22.064890 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 05 16:23:22.223977 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:22.223940 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:23:22.224412 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:22.224375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" event={"ID":"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec","Type":"ContainerStarted","Data":"27b08dd895f172d73805e6ad2de8a871109dde6e712484d7846707173dc9349b"} Aug 05 16:23:22.300968 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:22.300935 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:22.401374 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:22.401340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:23.030106 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:23.030077 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:23.091644 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:23.091613 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:23.114771 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:23.114739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:23.114771 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:23.114765 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:23.114992 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:23.114738 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:23.114992 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:23.114899 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:23.115074 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:23.115003 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:23.115074 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:23.115064 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:23.228277 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:23.228244 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" event={"ID":"9edbc7ed-36b5-4170-8f45-d267b70eac99","Type":"ContainerStarted","Data":"86d45089d3ac2fa0df2c1410ba2be864b5c6f264d2248d55dc5a1aa864c78011"} Aug 05 16:23:23.248494 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:23.248435 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8sqf" podStartSLOduration=7.424509387 podStartE2EDuration="25.248420946s" totalImagesPullingTime="17.823911559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:23.245565746 +0000 UTC m=+26.604817592" watchObservedRunningTime="2026-08-05 16:23:23.248420946 +0000 UTC m=+26.607672789" Aug 05 16:23:23.248645 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:23.248570 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-17.ec2.internal" podStartSLOduration=25.24856483 podStartE2EDuration="25.24856483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:21.24517805 +0000 UTC m=+24.604429896" watchObservedRunningTime="2026-08-05 16:23:23.24856483 +0000 UTC m=+26.607816673" Aug 05 16:23:23.724139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:23.724102 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-559j8" Aug 05 16:23:23.724687 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:23.724662 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-559j8" Aug 05 16:23:24.231142 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:24.231116 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-559j8" Aug 05 16:23:24.231572 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:24.231410 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-559j8" Aug 05 16:23:24.298390 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:24.298348 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:25.117216 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:25.117190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:25.117216 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:25.117204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:25.117445 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:25.117190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:25.117445 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:25.117295 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:25.117445 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:25.117389 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:25.117602 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:25.117464 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:26.236372 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:26.236198 2575 generic.go:356] "Generic (PLEG): container finished" podID="6bd12d48-f8bc-4866-ac63-a9b61e696eb0" containerID="54896a9dc0f40880387718fbd81a00f93e374151ef9678ad049454ac76a2e1da" exitCode=0 Aug 05 16:23:26.237118 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:26.236296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" event={"ID":"6bd12d48-f8bc-4866-ac63-a9b61e696eb0","Type":"ContainerDied","Data":"54896a9dc0f40880387718fbd81a00f93e374151ef9678ad049454ac76a2e1da"} Aug 05 16:23:26.239534 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:26.239517 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:23:26.239916 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:26.239895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" event={"ID":"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec","Type":"ContainerStarted","Data":"07ed15a440922ffb42e412c678144b581f83a912be4dd2a9bd6f44d15ffd147e"} Aug 05 16:23:26.240160 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:26.240146 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:23:26.240210 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:26.240202 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:23:26.240260 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:26.240242 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:23:26.240260 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:26.240250 2575 scope.go:117] "RemoveContainer" containerID="d1b62bad08ac9b3ce7065cb00be08130bd0fcfdb74d709d858ac7d315afb41dc" Aug 05 16:23:26.257291 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:26.257262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:23:26.257510 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:26.257497 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:23:27.114879 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.114650 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:27.115020 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.114719 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:27.115020 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:27.115004 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:27.115108 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:27.114892 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:27.115108 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.114742 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:27.115162 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:27.115111 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:27.243758 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.243729 2575 generic.go:356] "Generic (PLEG): container finished" podID="6bd12d48-f8bc-4866-ac63-a9b61e696eb0" containerID="2d487bbdb9dde8e43f09b7fb101ba88b353f38a9d6165a67b6e15d3858c4b053" exitCode=0 Aug 05 16:23:27.244144 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.243822 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" event={"ID":"6bd12d48-f8bc-4866-ac63-a9b61e696eb0","Type":"ContainerDied","Data":"2d487bbdb9dde8e43f09b7fb101ba88b353f38a9d6165a67b6e15d3858c4b053"} Aug 05 16:23:27.247521 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.247507 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:23:27.247890 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.247856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" event={"ID":"c4f66f40-5075-4bbd-9f6d-95c6f63f00ec","Type":"ContainerStarted","Data":"553b9679d3d5fedfcfc832a1a70ae8fa8123c467ef99180ffb92b080114f93fe"} Aug 05 16:23:27.296288 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.296244 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" podStartSLOduration=9.58984235 podStartE2EDuration="29.296229112s" totalImagesPullingTime="19.706386762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:27.29621566 +0000 UTC m=+30.655467504" watchObservedRunningTime="2026-08-05 16:23:27.296229112 +0000 UTC m=+30.655480937" Aug 05 16:23:27.779888 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.779851 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c5l2j"] Aug 05 16:23:27.780056 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.780031 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:27.780168 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:27.780139 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:27.782876 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.782724 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jff5g"] Aug 05 16:23:27.782876 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.782857 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:27.783063 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:27.783002 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:27.794310 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.794238 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-57h6j"] Aug 05 16:23:27.794422 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:27.794360 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:27.794498 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:27.794458 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:28.208255 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:28.208209 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:28.252559 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:28.252528 2575 generic.go:356] "Generic (PLEG): container finished" podID="6bd12d48-f8bc-4866-ac63-a9b61e696eb0" containerID="73e7db4c174b25d5231c2b223fcf2257349a725594568409a13cbb8ac17a3218" exitCode=0 Aug 05 16:23:28.252960 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:28.252619 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" event={"ID":"6bd12d48-f8bc-4866-ac63-a9b61e696eb0","Type":"ContainerDied","Data":"73e7db4c174b25d5231c2b223fcf2257349a725594568409a13cbb8ac17a3218"} Aug 05 16:23:28.426085 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:28.426011 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:28.525919 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:28.525888 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:29.114875 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:29.114828 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:29.115060 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:29.115007 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:29.153378 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:29.153340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:29.226731 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:29.226677 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:29.431545 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:29.431458 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:29.488461 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:29.488411 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:29.593986 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:29.593956 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:29.664085 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:29.664032 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:29.722290 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:29.722141 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:30.114480 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:30.114401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:30.114650 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:30.114408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:30.114650 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:30.114621 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5l2j" podUID="8ddce42c-0420-4820-9d02-f1bfbdd3bf42" Aug 05 16:23:30.114650 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:30.114502 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57h6j" podUID="25e06ceb-1b89-4317-8789-90457b17cb83" Aug 05 16:23:30.130767 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:30.130708 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:30.229195 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:30.229140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:30.507833 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:30.507741 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-22g2w_d3c6ce5d-ec80-437e-a440-8cc72d6de221/dns-node-resolver/0.log" Aug 05 16:23:30.752032 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:30.751993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:30.752209 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:30.752082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5vc\" (UniqueName: \"kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc\") pod \"network-check-target-57h6j\" (UID: \"25e06ceb-1b89-4317-8789-90457b17cb83\") " pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:30.752209 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:30.752127 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:30.752209 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:30.752192 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs podName:b8676cf2-b5e7-47f2-895e-f5271300ea80 nodeName:}" failed. No retries permitted until 2026-08-05 16:24:02.752173348 +0000 UTC m=+66.111425173 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs") pod "network-metrics-daemon-jff5g" (UID: "b8676cf2-b5e7-47f2-895e-f5271300ea80") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:30.752306 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:30.752219 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:23:30.752306 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:30.752241 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:23:30.752306 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:30.752255 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2x5vc for pod openshift-network-diagnostics/network-check-target-57h6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:30.752306 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:30.752301 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc podName:25e06ceb-1b89-4317-8789-90457b17cb83 nodeName:}" failed. No retries permitted until 2026-08-05 16:24:02.752289939 +0000 UTC m=+66.111541765 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x5vc" (UniqueName: "kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc") pod "network-check-target-57h6j" (UID: "25e06ceb-1b89-4317-8789-90457b17cb83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:30.837985 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:30.837892 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:30.852948 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:30.852924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:30.853099 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:30.853046 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:30.853099 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:30.853096 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret podName:8ddce42c-0420-4820-9d02-f1bfbdd3bf42 nodeName:}" failed. No retries permitted until 2026-08-05 16:24:02.853082466 +0000 UTC m=+66.212334289 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret") pod "global-pull-secret-syncer-c5l2j" (UID: "8ddce42c-0420-4820-9d02-f1bfbdd3bf42") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:30.942098 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:30.942068 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:31.116002 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:31.115927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:31.116151 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:31.116055 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jff5g" podUID="b8676cf2-b5e7-47f2-895e-f5271300ea80" Aug 05 16:23:31.907959 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:31.907931 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-stg9n_4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3/node-ca/0.log" Aug 05 16:23:32.016440 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.016408 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-17.ec2.internal" event="NodeReady" Aug 05 16:23:32.016596 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.016536 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 05 16:23:32.061825 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.061737 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d59b8"] Aug 05 16:23:32.082226 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.082188 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s4ndk"] Aug 05 16:23:32.082922 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.082897 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.085446 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.085421 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 05 16:23:32.085564 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.085462 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 05 16:23:32.085719 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.085703 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-422vm\"" Aug 05 16:23:32.085788 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.085716 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 05 16:23:32.086560 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.086543 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 05 16:23:32.094253 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.094234 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d59b8"] Aug 05 16:23:32.094339 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.094270 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s4ndk"] Aug 05 16:23:32.094378 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.094359 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s4ndk" Aug 05 16:23:32.097052 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.097035 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 05 16:23:32.097154 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.097132 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-chrqx\"" Aug 05 16:23:32.097154 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.097135 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 05 16:23:32.097269 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.097248 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 05 16:23:32.114645 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.114625 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:23:32.114645 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.114642 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:23:32.118340 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.118145 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 05 16:23:32.118340 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.118159 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 16:23:32.118340 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.118161 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t7dww\"" Aug 05 16:23:32.118517 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.118443 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 05 16:23:32.161499 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.161477 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b7drl"] Aug 05 16:23:32.180145 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.180087 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b7drl"] Aug 05 16:23:32.180258 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.180174 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.182728 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.182712 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 05 16:23:32.182940 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.182923 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 05 16:23:32.183039 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.182940 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w85qf\"" Aug 05 16:23:32.256503 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.256474 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:32.261241 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/049c9ec1-4759-426e-9290-5be491ba3f71-config-volume\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.261358 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f7aff24-8650-49c3-99a6-d2c7cce19769-data-volume\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.261358 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2234a147-bfa6-4060-90f2-83899121eadd-cert\") pod \"ingress-canary-s4ndk\" (UID: \"2234a147-bfa6-4060-90f2-83899121eadd\") " pod="openshift-ingress-canary/ingress-canary-s4ndk" Aug 05 16:23:32.261469 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.261469 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049c9ec1-4759-426e-9290-5be491ba3f71-tmp-dir\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.261469 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.261469 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4sch\" (UniqueName: \"kubernetes.io/projected/2234a147-bfa6-4060-90f2-83899121eadd-kube-api-access-l4sch\") pod \"ingress-canary-s4ndk\" (UID: \"2234a147-bfa6-4060-90f2-83899121eadd\") " pod="openshift-ingress-canary/ingress-canary-s4ndk" Aug 05 16:23:32.261673 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261477 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.261673 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xnr\" (UniqueName: \"kubernetes.io/projected/049c9ec1-4759-426e-9290-5be491ba3f71-kube-api-access-x6xnr\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.261673 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f7aff24-8650-49c3-99a6-d2c7cce19769-crio-socket\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.261673 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.261599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56jbf\" (UniqueName: \"kubernetes.io/projected/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-api-access-56jbf\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.308039 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.308015 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:32.362128 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xnr\" (UniqueName: \"kubernetes.io/projected/049c9ec1-4759-426e-9290-5be491ba3f71-kube-api-access-x6xnr\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.362128 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f7aff24-8650-49c3-99a6-d2c7cce19769-crio-socket\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.362128 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56jbf\" (UniqueName: \"kubernetes.io/projected/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-api-access-56jbf\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.362321 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f7aff24-8650-49c3-99a6-d2c7cce19769-crio-socket\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.362358 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/049c9ec1-4759-426e-9290-5be491ba3f71-config-volume\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.362358 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f7aff24-8650-49c3-99a6-d2c7cce19769-data-volume\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.362425 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2234a147-bfa6-4060-90f2-83899121eadd-cert\") pod \"ingress-canary-s4ndk\" (UID: \"2234a147-bfa6-4060-90f2-83899121eadd\") " pod="openshift-ingress-canary/ingress-canary-s4ndk" Aug 05 16:23:32.362425 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.362425 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049c9ec1-4759-426e-9290-5be491ba3f71-tmp-dir\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.362521 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.362521 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sch\" (UniqueName: \"kubernetes.io/projected/2234a147-bfa6-4060-90f2-83899121eadd-kube-api-access-l4sch\") pod \"ingress-canary-s4ndk\" (UID: \"2234a147-bfa6-4060-90f2-83899121eadd\") " pod="openshift-ingress-canary/ingress-canary-s4ndk" Aug 05 16:23:32.362521 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.362618 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.362575 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:32.362618 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.362615 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls podName:7f7aff24-8650-49c3-99a6-d2c7cce19769 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:32.862602788 +0000 UTC m=+36.221854610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d59b8" (UID: "7f7aff24-8650-49c3-99a6-d2c7cce19769") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:32.362717 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362699 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/049c9ec1-4759-426e-9290-5be491ba3f71-config-volume\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.362836 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.362824 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:32.362886 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.362868 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls podName:049c9ec1-4759-426e-9290-5be491ba3f71 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:32.862855108 +0000 UTC m=+36.222106932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls") pod "dns-default-b7drl" (UID: "049c9ec1-4759-426e-9290-5be491ba3f71") : secret "dns-default-metrics-tls" not found Aug 05 16:23:32.362984 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.362964 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f7aff24-8650-49c3-99a6-d2c7cce19769-data-volume\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.363053 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.363042 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:23:32.363116 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.363105 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm podName:7f7aff24-8650-49c3-99a6-d2c7cce19769 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:32.863079737 +0000 UTC m=+36.222331582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d59b8" (UID: "7f7aff24-8650-49c3-99a6-d2c7cce19769") : configmap "kube-rbac-proxy" not found Aug 05 16:23:32.363116 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.363102 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049c9ec1-4759-426e-9290-5be491ba3f71-tmp-dir\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.367826 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.367674 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2234a147-bfa6-4060-90f2-83899121eadd-cert\") pod \"ingress-canary-s4ndk\" (UID: \"2234a147-bfa6-4060-90f2-83899121eadd\") " pod="openshift-ingress-canary/ingress-canary-s4ndk" Aug 05 16:23:32.371649 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.371629 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xnr\" (UniqueName: \"kubernetes.io/projected/049c9ec1-4759-426e-9290-5be491ba3f71-kube-api-access-x6xnr\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.371761 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.371696 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56jbf\" (UniqueName: \"kubernetes.io/projected/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-api-access-56jbf\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.372307 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.372287 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sch\" (UniqueName: \"kubernetes.io/projected/2234a147-bfa6-4060-90f2-83899121eadd-kube-api-access-l4sch\") pod \"ingress-canary-s4ndk\" (UID: \"2234a147-bfa6-4060-90f2-83899121eadd\") " pod="openshift-ingress-canary/ingress-canary-s4ndk" Aug 05 16:23:32.394352 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.394321 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:32.404157 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.404140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s4ndk" Aug 05 16:23:32.436350 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.436333 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-kqwq9"] Aug 05 16:23:32.448751 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.448725 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-kqwq9"] Aug 05 16:23:32.448900 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.448858 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.451979 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.451960 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 05 16:23:32.452202 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.452186 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 05 16:23:32.452396 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.452378 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 05 16:23:32.452701 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.452680 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 05 16:23:32.452842 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.452768 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2vh5s\"" Aug 05 16:23:32.468165 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.468145 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:32.528111 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.528084 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:32.564179 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.564141 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s5k5\" (UniqueName: \"kubernetes.io/projected/90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb-kube-api-access-7s5k5\") pod \"service-ca-98c88fbb5-kqwq9\" (UID: \"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb\") " pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.564328 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.564286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb-signing-cabundle\") pod \"service-ca-98c88fbb5-kqwq9\" (UID: \"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb\") " pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.564389 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.564323 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb-signing-key\") pod \"service-ca-98c88fbb5-kqwq9\" (UID: \"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb\") " pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.665093 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.665021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s5k5\" (UniqueName: \"kubernetes.io/projected/90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb-kube-api-access-7s5k5\") pod \"service-ca-98c88fbb5-kqwq9\" (UID: \"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb\") " pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.665254 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.665100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb-signing-cabundle\") pod \"service-ca-98c88fbb5-kqwq9\" (UID: \"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb\") " pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.665254 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.665132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb-signing-key\") pod \"service-ca-98c88fbb5-kqwq9\" (UID: \"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb\") " pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.665878 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.665856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb-signing-cabundle\") pod \"service-ca-98c88fbb5-kqwq9\" (UID: \"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb\") " pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.667983 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.667963 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb-signing-key\") pod \"service-ca-98c88fbb5-kqwq9\" (UID: \"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb\") " pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.673995 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.673971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s5k5\" (UniqueName: \"kubernetes.io/projected/90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb-kube-api-access-7s5k5\") pod \"service-ca-98c88fbb5-kqwq9\" (UID: \"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb\") " pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.760381 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.760348 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" Aug 05 16:23:32.866207 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.866172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.866354 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.866234 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:32.866354 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.866288 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:23:32.866354 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:32.866316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:32.866474 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.866359 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm podName:7f7aff24-8650-49c3-99a6-d2c7cce19769 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:33.866337086 +0000 UTC m=+37.225588947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d59b8" (UID: "7f7aff24-8650-49c3-99a6-d2c7cce19769") : configmap "kube-rbac-proxy" not found Aug 05 16:23:32.866474 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.866434 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:32.866474 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.866470 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls podName:049c9ec1-4759-426e-9290-5be491ba3f71 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:33.86645919 +0000 UTC m=+37.225711012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls") pod "dns-default-b7drl" (UID: "049c9ec1-4759-426e-9290-5be491ba3f71") : secret "dns-default-metrics-tls" not found Aug 05 16:23:32.866588 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.866512 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:32.866588 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:32.866533 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls podName:7f7aff24-8650-49c3-99a6-d2c7cce19769 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:33.866525923 +0000 UTC m=+37.225777745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d59b8" (UID: "7f7aff24-8650-49c3-99a6-d2c7cce19769") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:33.114710 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:33.114340 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:23:33.119876 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:33.119161 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zm8j\"" Aug 05 16:23:33.119876 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:33.119538 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 16:23:33.251293 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:33.251265 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-kqwq9"] Aug 05 16:23:33.257247 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:33.257226 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s4ndk"] Aug 05 16:23:33.271360 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:23:33.271333 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e3e95c_df0c_4aa8_b01f_80a69ffbe7eb.slice/crio-81569fd9fb573b27d9c9df15026a7ae166768ad6d0b131a54efe5995f059fe17 WatchSource:0}: Error finding container 81569fd9fb573b27d9c9df15026a7ae166768ad6d0b131a54efe5995f059fe17: Status 404 returned error can't find the container with id 81569fd9fb573b27d9c9df15026a7ae166768ad6d0b131a54efe5995f059fe17 Aug 05 16:23:33.271881 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:23:33.271863 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2234a147_bfa6_4060_90f2_83899121eadd.slice/crio-b29e60a357db23b3d0e839106a4c3ba9bfb46c971f5281e520c6f1e189d5969d WatchSource:0}: Error finding container b29e60a357db23b3d0e839106a4c3ba9bfb46c971f5281e520c6f1e189d5969d: Status 404 returned error can't find the container with id b29e60a357db23b3d0e839106a4c3ba9bfb46c971f5281e520c6f1e189d5969d Aug 05 16:23:33.664535 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:33.664475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:33.873452 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:33.873419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:33.873736 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:33.873467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:33.873736 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:33.873548 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:23:33.873736 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:33.873609 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:33.873736 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:33.873638 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm podName:7f7aff24-8650-49c3-99a6-d2c7cce19769 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:35.873615788 +0000 UTC m=+39.232867618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d59b8" (UID: "7f7aff24-8650-49c3-99a6-d2c7cce19769") : configmap "kube-rbac-proxy" not found Aug 05 16:23:33.873736 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:33.873605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:33.873736 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:33.873662 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls podName:049c9ec1-4759-426e-9290-5be491ba3f71 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:35.873652287 +0000 UTC m=+39.232904110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls") pod "dns-default-b7drl" (UID: "049c9ec1-4759-426e-9290-5be491ba3f71") : secret "dns-default-metrics-tls" not found Aug 05 16:23:33.873736 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:33.873719 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:33.874040 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:33.873766 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls podName:7f7aff24-8650-49c3-99a6-d2c7cce19769 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:35.873748595 +0000 UTC m=+39.233000418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d59b8" (UID: "7f7aff24-8650-49c3-99a6-d2c7cce19769") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:34.268886 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:34.268851 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s4ndk" event={"ID":"2234a147-bfa6-4060-90f2-83899121eadd","Type":"ContainerStarted","Data":"b29e60a357db23b3d0e839106a4c3ba9bfb46c971f5281e520c6f1e189d5969d"} Aug 05 16:23:34.270836 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:34.270766 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" event={"ID":"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb","Type":"ContainerStarted","Data":"81569fd9fb573b27d9c9df15026a7ae166768ad6d0b131a54efe5995f059fe17"} Aug 05 16:23:34.274269 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:34.274232 2575 generic.go:356] "Generic (PLEG): container finished" podID="6bd12d48-f8bc-4866-ac63-a9b61e696eb0" containerID="c60b6e3577c8699515783589e57d22c3536e7f7018bc6baa524a4a7fbd012dbf" exitCode=0 Aug 05 16:23:34.274269 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:34.274268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" event={"ID":"6bd12d48-f8bc-4866-ac63-a9b61e696eb0","Type":"ContainerDied","Data":"c60b6e3577c8699515783589e57d22c3536e7f7018bc6baa524a4a7fbd012dbf"} Aug 05 16:23:35.280417 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:35.280239 2575 generic.go:356] "Generic (PLEG): container finished" podID="6bd12d48-f8bc-4866-ac63-a9b61e696eb0" containerID="7c23c823c69005c4843c9d1bcba23c676441d365a27ef2b12512bf549490dc1d" exitCode=0 Aug 05 16:23:35.280747 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:35.280313 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" event={"ID":"6bd12d48-f8bc-4866-ac63-a9b61e696eb0","Type":"ContainerDied","Data":"7c23c823c69005c4843c9d1bcba23c676441d365a27ef2b12512bf549490dc1d"} Aug 05 16:23:35.893735 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:35.893701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:35.893971 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:35.893776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:35.893971 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:35.893834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:35.893971 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:35.893893 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:35.893971 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:35.893895 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:35.893971 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:35.893900 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:23:35.893971 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:35.893968 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls podName:049c9ec1-4759-426e-9290-5be491ba3f71 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:39.893948449 +0000 UTC m=+43.253200289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls") pod "dns-default-b7drl" (UID: "049c9ec1-4759-426e-9290-5be491ba3f71") : secret "dns-default-metrics-tls" not found Aug 05 16:23:35.894295 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:35.893989 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls podName:7f7aff24-8650-49c3-99a6-d2c7cce19769 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:39.893977859 +0000 UTC m=+43.253229690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d59b8" (UID: "7f7aff24-8650-49c3-99a6-d2c7cce19769") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:35.894295 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:35.894003 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm podName:7f7aff24-8650-49c3-99a6-d2c7cce19769 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:39.89399539 +0000 UTC m=+43.253247213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d59b8" (UID: "7f7aff24-8650-49c3-99a6-d2c7cce19769") : configmap "kube-rbac-proxy" not found Aug 05 16:23:36.404502 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:36.404478 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:37.289357 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.289321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" event={"ID":"6bd12d48-f8bc-4866-ac63-a9b61e696eb0","Type":"ContainerStarted","Data":"b6c5dcab07c6fcaa60e721d677dcbdab981fbf2c298825645065821007fd50d6"} Aug 05 16:23:37.290703 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.290677 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s4ndk" event={"ID":"2234a147-bfa6-4060-90f2-83899121eadd","Type":"ContainerStarted","Data":"0e3c592d186c79fab38ed2145a7f32f9b8d3199526047941d8d9804a693a0e08"} Aug 05 16:23:37.291956 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.291933 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" event={"ID":"90e3e95c-df0c-4aa8-b01f-80a69ffbe7eb","Type":"ContainerStarted","Data":"4f6cdcf13491569311bcc2c6a40038777cdb94a5a4c0b045faa9aef4dee0c59d"} Aug 05 16:23:37.319687 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.319646 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t2jzm" podStartSLOduration=7.887076959 podStartE2EDuration="39.319631934s" totalImagesPullingTime="31.432554975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:37.319066579 +0000 UTC m=+40.678318422" watchObservedRunningTime="2026-08-05 16:23:37.319631934 +0000 UTC m=+40.678883791" Aug 05 16:23:37.338000 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.337946 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-kqwq9" podStartSLOduration=2.327288229 podStartE2EDuration="5.337930604s" totalImagesPullingTime="3.010642375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:37.337008011 +0000 UTC m=+40.696259856" watchObservedRunningTime="2026-08-05 16:23:37.337930604 +0000 UTC m=+40.697182448" Aug 05 16:23:37.353371 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.353335 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s4ndk" podStartSLOduration=2.336626524 podStartE2EDuration="5.353323257s" totalImagesPullingTime="3.016696733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:37.353108164 +0000 UTC m=+40.712360011" watchObservedRunningTime="2026-08-05 16:23:37.353323257 +0000 UTC m=+40.712575101" Aug 05 16:23:37.354914 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.354895 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:37.409499 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.409442 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:37.623494 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.623462 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:37.855170 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.855137 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:37.955620 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:37.955545 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:38.056391 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:38.056369 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:38.187943 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:38.187876 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:38.297007 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:38.296972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:38.356522 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:38.356492 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:38.968527 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:38.968504 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:39.518000 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:39.517972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:39.713881 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:39.713850 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:39.921475 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:39.921415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:39.921475 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:39.921458 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:39.921619 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:39.921526 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:23:39.921619 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:39.921584 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm podName:7f7aff24-8650-49c3-99a6-d2c7cce19769 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:47.921568368 +0000 UTC m=+51.280820195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d59b8" (UID: "7f7aff24-8650-49c3-99a6-d2c7cce19769") : configmap "kube-rbac-proxy" not found Aug 05 16:23:39.921619 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:39.921612 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:39.921716 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:39.921651 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls podName:7f7aff24-8650-49c3-99a6-d2c7cce19769 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:47.921639327 +0000 UTC m=+51.280891149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d59b8" (UID: "7f7aff24-8650-49c3-99a6-d2c7cce19769") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:39.921716 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:39.921705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:39.921780 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:39.921759 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:39.921836 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:39.921786 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls podName:049c9ec1-4759-426e-9290-5be491ba3f71 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:47.921779448 +0000 UTC m=+51.281031271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls") pod "dns-default-b7drl" (UID: "049c9ec1-4759-426e-9290-5be491ba3f71") : secret "dns-default-metrics-tls" not found Aug 05 16:23:40.135637 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:40.135606 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:40.319555 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:40.319525 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:40.792431 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:40.792405 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:40.911879 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:40.911856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:41.360308 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:41.360277 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:41.481013 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:41.480990 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:41.538000 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:41.537977 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:42.118129 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:42.118099 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:42.249614 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:42.249581 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:42.349573 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:42.349545 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:42.719561 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:42.719524 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:42.917893 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:42.917840 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:43.116639 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:43.116601 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:43.802092 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:43.802059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:43.914156 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:43.914120 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:44.459983 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:44.459946 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:44.714609 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:44.714514 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:45.111651 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:45.111617 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:45.319681 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:45.319651 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:45.712363 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:45.712324 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:46.316848 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:46.316761 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:46.711025 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:46.710935 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:47.133944 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:47.133902 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:47.346480 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:47.346449 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:47.916416 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:47.916384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:47.979737 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:47.979710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:47.979893 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:47.979748 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:47.979893 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:47.979811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:47.980358 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:47.980339 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f7aff24-8650-49c3-99a6-d2c7cce19769-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:47.983699 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:47.983676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/049c9ec1-4759-426e-9290-5be491ba3f71-metrics-tls\") pod \"dns-default-b7drl\" (UID: \"049c9ec1-4759-426e-9290-5be491ba3f71\") " pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:47.983830 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:47.983676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f7aff24-8650-49c3-99a6-d2c7cce19769-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d59b8\" (UID: \"7f7aff24-8650-49c3-99a6-d2c7cce19769\") " pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:47.993641 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:47.993620 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d59b8" Aug 05 16:23:48.090094 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:48.090068 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:48.118960 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:48.118930 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:48.131161 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:48.131125 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d59b8"] Aug 05 16:23:48.134055 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:23:48.134033 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7aff24_8650_49c3_99a6_d2c7cce19769.slice/crio-dd76299d7998a10d78c8fe87cc20622fe50eaa225021e3c36d86d7f2aa893b84 WatchSource:0}: Error finding container dd76299d7998a10d78c8fe87cc20622fe50eaa225021e3c36d86d7f2aa893b84: Status 404 returned error can't find the container with id dd76299d7998a10d78c8fe87cc20622fe50eaa225021e3c36d86d7f2aa893b84 Aug 05 16:23:48.219514 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:48.219490 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:48.255343 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:48.255316 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b7drl"] Aug 05 16:23:48.258328 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:23:48.258305 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049c9ec1_4759_426e_9290_5be491ba3f71.slice/crio-cc48fb182221ae5a4ba227c17604056ad6200a51d14244c6a871386376939df7 WatchSource:0}: Error finding container cc48fb182221ae5a4ba227c17604056ad6200a51d14244c6a871386376939df7: Status 404 returned error can't find the container with id cc48fb182221ae5a4ba227c17604056ad6200a51d14244c6a871386376939df7 Aug 05 16:23:48.291240 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:48.291211 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:48.320254 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:48.320219 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7drl" event={"ID":"049c9ec1-4759-426e-9290-5be491ba3f71","Type":"ContainerStarted","Data":"cc48fb182221ae5a4ba227c17604056ad6200a51d14244c6a871386376939df7"} Aug 05 16:23:48.321629 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:48.321609 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d59b8" event={"ID":"7f7aff24-8650-49c3-99a6-d2c7cce19769","Type":"ContainerStarted","Data":"12631341ea66e9850e8adcf3d2ce9dc57526752b16b8ae37d4f0f3dcfff60f53"} Aug 05 16:23:48.321742 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:48.321649 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d59b8" event={"ID":"7f7aff24-8650-49c3-99a6-d2c7cce19769","Type":"ContainerStarted","Data":"dd76299d7998a10d78c8fe87cc20622fe50eaa225021e3c36d86d7f2aa893b84"} Aug 05 16:23:48.344707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:48.344678 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:50.330416 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:50.330370 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d59b8" event={"ID":"7f7aff24-8650-49c3-99a6-d2c7cce19769","Type":"ContainerStarted","Data":"1733e252f7137a43c83ba9ba8022267a357823886ca3a8e1018786627f88442f"} Aug 05 16:23:51.074878 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.074848 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m"] Aug 05 16:23:51.077042 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.077020 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" Aug 05 16:23:51.080134 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.080104 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 05 16:23:51.080363 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.080346 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Aug 05 16:23:51.081272 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.081247 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ng7pd\"" Aug 05 16:23:51.081826 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.081509 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 05 16:23:51.081826 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.081674 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 05 16:23:51.092694 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.092665 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m"] Aug 05 16:23:51.100158 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.100131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16705b9d-1f20-4c72-b48c-e98fe2cbe291-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5648d879c8-bf77m\" (UID: \"16705b9d-1f20-4c72-b48c-e98fe2cbe291\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" Aug 05 16:23:51.100304 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.100177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8v4l\" (UniqueName: \"kubernetes.io/projected/16705b9d-1f20-4c72-b48c-e98fe2cbe291-kube-api-access-c8v4l\") pod \"managed-serviceaccount-addon-agent-5648d879c8-bf77m\" (UID: \"16705b9d-1f20-4c72-b48c-e98fe2cbe291\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" Aug 05 16:23:51.126685 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.126658 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:51.169670 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.169634 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc"] Aug 05 16:23:51.172143 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.172118 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6ff4748cdb-cslpc"] Aug 05 16:23:51.172252 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.172207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" Aug 05 16:23:51.174086 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.174068 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.174999 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.174977 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2sc2j\"" Aug 05 16:23:51.175122 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.175032 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Aug 05 16:23:51.175122 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.175060 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Aug 05 16:23:51.178075 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.178058 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 05 16:23:51.178513 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.178464 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 05 16:23:51.178613 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.178518 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kcq9m\"" Aug 05 16:23:51.179038 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.179020 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 05 16:23:51.185558 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.185533 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc"] Aug 05 16:23:51.189395 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.188494 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 05 16:23:51.201960 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.201778 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc887723-951c-470f-99ce-7684edad114b-registry-certificates\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.201960 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.201835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc887723-951c-470f-99ce-7684edad114b-trusted-ca\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.201960 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.201872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc887723-951c-470f-99ce-7684edad114b-ca-trust-extracted\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.202204 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.201980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjp4m\" (UniqueName: \"kubernetes.io/projected/bc887723-951c-470f-99ce-7684edad114b-kube-api-access-bjp4m\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.202204 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.202036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16705b9d-1f20-4c72-b48c-e98fe2cbe291-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5648d879c8-bf77m\" (UID: \"16705b9d-1f20-4c72-b48c-e98fe2cbe291\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" Aug 05 16:23:51.202204 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.202062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc887723-951c-470f-99ce-7684edad114b-image-registry-private-configuration\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.202204 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.202107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8v4l\" (UniqueName: \"kubernetes.io/projected/16705b9d-1f20-4c72-b48c-e98fe2cbe291-kube-api-access-c8v4l\") pod \"managed-serviceaccount-addon-agent-5648d879c8-bf77m\" (UID: \"16705b9d-1f20-4c72-b48c-e98fe2cbe291\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" Aug 05 16:23:51.202204 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.202129 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc887723-951c-470f-99ce-7684edad114b-registry-tls\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.202204 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.202153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc887723-951c-470f-99ce-7684edad114b-installation-pull-secrets\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.202204 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.202173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc887723-951c-470f-99ce-7684edad114b-bound-sa-token\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.202204 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.202197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07165f9c-e008-4e15-bf44-24e63eb41f3b-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-zqvmc\" (UID: \"07165f9c-e008-4e15-bf44-24e63eb41f3b\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" Aug 05 16:23:51.202562 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.202217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07165f9c-e008-4e15-bf44-24e63eb41f3b-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zqvmc\" (UID: \"07165f9c-e008-4e15-bf44-24e63eb41f3b\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" Aug 05 16:23:51.204103 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.204074 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ff4748cdb-cslpc"] Aug 05 16:23:51.205380 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.205355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16705b9d-1f20-4c72-b48c-e98fe2cbe291-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5648d879c8-bf77m\" (UID: \"16705b9d-1f20-4c72-b48c-e98fe2cbe291\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" Aug 05 16:23:51.206260 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.206238 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:51.212155 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.212130 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8v4l\" (UniqueName: \"kubernetes.io/projected/16705b9d-1f20-4c72-b48c-e98fe2cbe291-kube-api-access-c8v4l\") pod \"managed-serviceaccount-addon-agent-5648d879c8-bf77m\" (UID: \"16705b9d-1f20-4c72-b48c-e98fe2cbe291\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" Aug 05 16:23:51.292314 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.292281 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:51.302968 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.302949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjp4m\" (UniqueName: \"kubernetes.io/projected/bc887723-951c-470f-99ce-7684edad114b-kube-api-access-bjp4m\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.303070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.303013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc887723-951c-470f-99ce-7684edad114b-image-registry-private-configuration\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.303070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.303050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc887723-951c-470f-99ce-7684edad114b-registry-tls\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.303180 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.303079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc887723-951c-470f-99ce-7684edad114b-installation-pull-secrets\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.303180 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.303134 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc887723-951c-470f-99ce-7684edad114b-bound-sa-token\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.303180 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.303159 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07165f9c-e008-4e15-bf44-24e63eb41f3b-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-zqvmc\" (UID: \"07165f9c-e008-4e15-bf44-24e63eb41f3b\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" Aug 05 16:23:51.303315 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.303266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07165f9c-e008-4e15-bf44-24e63eb41f3b-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zqvmc\" (UID: \"07165f9c-e008-4e15-bf44-24e63eb41f3b\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" Aug 05 16:23:51.303371 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.303318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc887723-951c-470f-99ce-7684edad114b-registry-certificates\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.303371 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.303344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc887723-951c-470f-99ce-7684edad114b-trusted-ca\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.303463 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.303375 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc887723-951c-470f-99ce-7684edad114b-ca-trust-extracted\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.303864 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.303844 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07165f9c-e008-4e15-bf44-24e63eb41f3b-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-zqvmc\" (UID: \"07165f9c-e008-4e15-bf44-24e63eb41f3b\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" Aug 05 16:23:51.304110 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.304086 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc887723-951c-470f-99ce-7684edad114b-ca-trust-extracted\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.304554 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.304531 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc887723-951c-470f-99ce-7684edad114b-trusted-ca\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.304634 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.304548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc887723-951c-470f-99ce-7684edad114b-registry-certificates\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.305989 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.305961 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc887723-951c-470f-99ce-7684edad114b-image-registry-private-configuration\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.306401 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.306373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc887723-951c-470f-99ce-7684edad114b-installation-pull-secrets\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.306488 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.306449 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07165f9c-e008-4e15-bf44-24e63eb41f3b-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zqvmc\" (UID: \"07165f9c-e008-4e15-bf44-24e63eb41f3b\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" Aug 05 16:23:51.306686 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.306666 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc887723-951c-470f-99ce-7684edad114b-registry-tls\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.314048 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.314028 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjp4m\" (UniqueName: \"kubernetes.io/projected/bc887723-951c-470f-99ce-7684edad114b-kube-api-access-bjp4m\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.316045 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.316025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc887723-951c-470f-99ce-7684edad114b-bound-sa-token\") pod \"image-registry-6ff4748cdb-cslpc\" (UID: \"bc887723-951c-470f-99ce-7684edad114b\") " pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.335612 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.335554 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d59b8" event={"ID":"7f7aff24-8650-49c3-99a6-d2c7cce19769","Type":"ContainerStarted","Data":"169be07dcd5171dcf7c778cf468a80fb9aa6e1655e1a1115dbf374a105dd1f83"} Aug 05 16:23:51.337278 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.337259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7drl" event={"ID":"049c9ec1-4759-426e-9290-5be491ba3f71","Type":"ContainerStarted","Data":"6e2d6ebc52c328d13ea8c4024099879efb8153a1cb10aa4295370bfe8d9dfc2f"} Aug 05 16:23:51.337360 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.337303 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7drl" event={"ID":"049c9ec1-4759-426e-9290-5be491ba3f71","Type":"ContainerStarted","Data":"eabae04bf67e8058fee9ff6b9145d7e1f11903fddda605ff30f10ea9f9db1b52"} Aug 05 16:23:51.337360 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.337342 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b7drl" Aug 05 16:23:51.357438 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.357390 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d59b8" podStartSLOduration=16.488205245 podStartE2EDuration="19.357372679s" totalImagesPullingTime="2.869167434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:23:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:51.357012179 +0000 UTC m=+54.716264028" watchObservedRunningTime="2026-08-05 16:23:51.357372679 +0000 UTC m=+54.716624525" Aug 05 16:23:51.373035 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.373012 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:51.383245 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.383202 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b7drl" podStartSLOduration=17.35626696 podStartE2EDuration="19.38318618s" totalImagesPullingTime="2.02691922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:51.383069882 +0000 UTC m=+54.742321726" watchObservedRunningTime="2026-08-05 16:23:51.38318618 +0000 UTC m=+54.742438027" Aug 05 16:23:51.395161 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.395136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" Aug 05 16:23:51.486134 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.486077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" Aug 05 16:23:51.494344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.494103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:51.521915 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.521889 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m"] Aug 05 16:23:51.524370 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:23:51.524269 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16705b9d_1f20_4c72_b48c_e98fe2cbe291.slice/crio-5344e13051f3c7a52b144436f32ca194db07cc0905ac67ef0783b539ede46da3 WatchSource:0}: Error finding container 5344e13051f3c7a52b144436f32ca194db07cc0905ac67ef0783b539ede46da3: Status 404 returned error can't find the container with id 5344e13051f3c7a52b144436f32ca194db07cc0905ac67ef0783b539ede46da3 Aug 05 16:23:51.638500 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.638472 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc"] Aug 05 16:23:51.641175 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:23:51.641148 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07165f9c_e008_4e15_bf44_24e63eb41f3b.slice/crio-c066a03fa02a585dc5b511d98dcef4904731b6821081007460920c084c773a60 WatchSource:0}: Error finding container c066a03fa02a585dc5b511d98dcef4904731b6821081007460920c084c773a60: Status 404 returned error can't find the container with id c066a03fa02a585dc5b511d98dcef4904731b6821081007460920c084c773a60 Aug 05 16:23:51.656913 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:51.656893 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ff4748cdb-cslpc"] Aug 05 16:23:51.659365 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:23:51.659345 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc887723_951c_470f_99ce_7684edad114b.slice/crio-2c6b609fadef0786ced63abd2009c230ce76ae548c6cf32c7c5161232ea391dc WatchSource:0}: Error finding container 2c6b609fadef0786ced63abd2009c230ce76ae548c6cf32c7c5161232ea391dc: Status 404 returned error can't find the container with id 2c6b609fadef0786ced63abd2009c230ce76ae548c6cf32c7c5161232ea391dc Aug 05 16:23:52.343379 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:52.343315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" event={"ID":"07165f9c-e008-4e15-bf44-24e63eb41f3b","Type":"ContainerStarted","Data":"c066a03fa02a585dc5b511d98dcef4904731b6821081007460920c084c773a60"} Aug 05 16:23:52.344679 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:52.344640 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" event={"ID":"16705b9d-1f20-4c72-b48c-e98fe2cbe291","Type":"ContainerStarted","Data":"5344e13051f3c7a52b144436f32ca194db07cc0905ac67ef0783b539ede46da3"} Aug 05 16:23:52.347654 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:52.347595 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" event={"ID":"bc887723-951c-470f-99ce-7684edad114b","Type":"ContainerStarted","Data":"2aedbc333323f562485a770cd581c01e8f73a97573bb0aaabd15b09b92d7dbb9"} Aug 05 16:23:52.347654 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:52.347638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" event={"ID":"bc887723-951c-470f-99ce-7684edad114b","Type":"ContainerStarted","Data":"2c6b609fadef0786ced63abd2009c230ce76ae548c6cf32c7c5161232ea391dc"} Aug 05 16:23:52.348419 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:52.348158 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:23:52.375306 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:52.375241 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" podStartSLOduration=1.375219244 podStartE2EDuration="1.375219244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:52.373600373 +0000 UTC m=+55.732852220" watchObservedRunningTime="2026-08-05 16:23:52.375219244 +0000 UTC m=+55.734471067" Aug 05 16:23:54.354166 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:54.354075 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" event={"ID":"16705b9d-1f20-4c72-b48c-e98fe2cbe291","Type":"ContainerStarted","Data":"1bdf1d2b5a08c9a21dd2200714e76bf35abf013332faaf8c0d6ee1e02251c233"} Aug 05 16:23:54.355536 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:54.355510 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" event={"ID":"07165f9c-e008-4e15-bf44-24e63eb41f3b","Type":"ContainerStarted","Data":"664940101a939f0316f51f5afcf786976c771fed4b25d79a16f99efbe8c4952d"} Aug 05 16:23:54.371696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:54.371652 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5648d879c8-bf77m" podStartSLOduration=0.858485617 podStartE2EDuration="3.371638338s" totalImagesPullingTime="2.513152721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:54.370132027 +0000 UTC m=+57.729383871" watchObservedRunningTime="2026-08-05 16:23:54.371638338 +0000 UTC m=+57.730890183" Aug 05 16:23:54.390162 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:54.390121 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zqvmc" podStartSLOduration=0.996572713 podStartE2EDuration="3.390105429s" totalImagesPullingTime="2.393532716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:54.389098248 +0000 UTC m=+57.748350093" watchObservedRunningTime="2026-08-05 16:23:54.390105429 +0000 UTC m=+57.749357274" Aug 05 16:23:54.517441 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:54.517402 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:54.617716 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:54.617638 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:57.504179 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:57.504139 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:57.665463 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:57.665434 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:57.668528 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:57.668505 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf"] Aug 05 16:23:57.671725 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:57.671706 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" Aug 05 16:23:57.674124 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:57.674103 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Aug 05 16:23:57.674231 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:57.674186 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sl9vp\"" Aug 05 16:23:57.679454 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:57.679429 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf"] Aug 05 16:23:57.750463 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:57.750431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29d89fc0-da14-4af8-9d43-affbb48319eb-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-v4dhf\" (UID: \"29d89fc0-da14-4af8-9d43-affbb48319eb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" Aug 05 16:23:57.851831 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:57.851733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29d89fc0-da14-4af8-9d43-affbb48319eb-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-v4dhf\" (UID: \"29d89fc0-da14-4af8-9d43-affbb48319eb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" Aug 05 16:23:57.851956 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:57.851867 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Aug 05 16:23:57.851956 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:23:57.851922 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29d89fc0-da14-4af8-9d43-affbb48319eb-tls-certificates podName:29d89fc0-da14-4af8-9d43-affbb48319eb nodeName:}" failed. No retries permitted until 2026-08-05 16:23:58.351908849 +0000 UTC m=+61.711160671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/29d89fc0-da14-4af8-9d43-affbb48319eb-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-v4dhf" (UID: "29d89fc0-da14-4af8-9d43-affbb48319eb") : secret "prometheus-operator-admission-webhook-tls" not found Aug 05 16:23:58.053689 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:58.053656 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:58.264231 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:58.264203 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7c4d" Aug 05 16:23:58.355176 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:58.355140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29d89fc0-da14-4af8-9d43-affbb48319eb-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-v4dhf\" (UID: \"29d89fc0-da14-4af8-9d43-affbb48319eb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" Aug 05 16:23:58.357747 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:58.357724 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29d89fc0-da14-4af8-9d43-affbb48319eb-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-v4dhf\" (UID: \"29d89fc0-da14-4af8-9d43-affbb48319eb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" Aug 05 16:23:58.383712 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:58.383679 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:58.482952 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:58.482924 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:58.580783 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:58.580689 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" Aug 05 16:23:58.705610 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:58.705578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf"] Aug 05 16:23:58.708447 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:23:58.708419 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d89fc0_da14_4af8_9d43_affbb48319eb.slice/crio-4f1cc555cfbd93f7f2fdcf4956c511eedabfa8398365257c0e8a68e8eabf41db WatchSource:0}: Error finding container 4f1cc555cfbd93f7f2fdcf4956c511eedabfa8398365257c0e8a68e8eabf41db: Status 404 returned error can't find the container with id 4f1cc555cfbd93f7f2fdcf4956c511eedabfa8398365257c0e8a68e8eabf41db Aug 05 16:23:59.371431 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:23:59.371399 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" event={"ID":"29d89fc0-da14-4af8-9d43-affbb48319eb","Type":"ContainerStarted","Data":"4f1cc555cfbd93f7f2fdcf4956c511eedabfa8398365257c0e8a68e8eabf41db"} Aug 05 16:24:00.375995 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.375918 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" event={"ID":"29d89fc0-da14-4af8-9d43-affbb48319eb","Type":"ContainerStarted","Data":"827bfa85e25ee8592a9371b7a33a6a5a8e7d700520e0b727bcb61ddeada61ef4"} Aug 05 16:24:00.376425 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.376130 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" Aug 05 16:24:00.380730 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.380708 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" Aug 05 16:24:00.391190 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.391153 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-v4dhf" podStartSLOduration=2.102756274 podStartE2EDuration="3.391137957s" totalImagesPullingTime="1.288381683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:00.390042894 +0000 UTC m=+63.749294737" watchObservedRunningTime="2026-08-05 16:24:00.391137957 +0000 UTC m=+63.750389806" Aug 05 16:24:00.695171 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.695083 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:00.735278 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.735248 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-gqnwh"] Aug 05 16:24:00.737683 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.737667 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.740781 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.740753 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 05 16:24:00.740963 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.740784 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Aug 05 16:24:00.740963 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.740752 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Aug 05 16:24:00.740963 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.740853 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 05 16:24:00.742028 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.742002 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8585r\"" Aug 05 16:24:00.742149 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.742044 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 05 16:24:00.748350 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.748326 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-gqnwh"] Aug 05 16:24:00.767624 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.767591 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:00.773835 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.773783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8575379f-09e6-4855-ae41-e804a5ee8a2f-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.773933 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.773910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8575379f-09e6-4855-ae41-e804a5ee8a2f-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.773998 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.773982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8575379f-09e6-4855-ae41-e804a5ee8a2f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.774076 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.774059 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgnw9\" (UniqueName: \"kubernetes.io/projected/8575379f-09e6-4855-ae41-e804a5ee8a2f-kube-api-access-cgnw9\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.875164 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.875130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8575379f-09e6-4855-ae41-e804a5ee8a2f-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.875296 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.875181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8575379f-09e6-4855-ae41-e804a5ee8a2f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.875296 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.875208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgnw9\" (UniqueName: \"kubernetes.io/projected/8575379f-09e6-4855-ae41-e804a5ee8a2f-kube-api-access-cgnw9\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.875397 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.875379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8575379f-09e6-4855-ae41-e804a5ee8a2f-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.875507 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:24:00.875489 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Aug 05 16:24:00.875561 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:24:00.875548 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8575379f-09e6-4855-ae41-e804a5ee8a2f-prometheus-operator-tls podName:8575379f-09e6-4855-ae41-e804a5ee8a2f nodeName:}" failed. No retries permitted until 2026-08-05 16:24:01.375532734 +0000 UTC m=+64.734784556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8575379f-09e6-4855-ae41-e804a5ee8a2f-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-gqnwh" (UID: "8575379f-09e6-4855-ae41-e804a5ee8a2f") : secret "prometheus-operator-tls" not found Aug 05 16:24:00.875948 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.875927 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8575379f-09e6-4855-ae41-e804a5ee8a2f-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.877883 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.877855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8575379f-09e6-4855-ae41-e804a5ee8a2f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:00.884172 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:00.884149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgnw9\" (UniqueName: \"kubernetes.io/projected/8575379f-09e6-4855-ae41-e804a5ee8a2f-kube-api-access-cgnw9\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:01.261219 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:01.261182 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:01.350604 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:01.350580 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b7drl" Aug 05 16:24:01.380286 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:01.380251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8575379f-09e6-4855-ae41-e804a5ee8a2f-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:01.382772 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:01.382746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8575379f-09e6-4855-ae41-e804a5ee8a2f-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-gqnwh\" (UID: \"8575379f-09e6-4855-ae41-e804a5ee8a2f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:01.646433 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:01.646336 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" Aug 05 16:24:01.772106 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:01.771951 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-gqnwh"] Aug 05 16:24:01.774749 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:24:01.774723 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8575379f_09e6_4855_ae41_e804a5ee8a2f.slice/crio-cd8ec94b1e08af27596f67d68c03ddbb7162dfc60f68454c09362870b1c23442 WatchSource:0}: Error finding container cd8ec94b1e08af27596f67d68c03ddbb7162dfc60f68454c09362870b1c23442: Status 404 returned error can't find the container with id cd8ec94b1e08af27596f67d68c03ddbb7162dfc60f68454c09362870b1c23442 Aug 05 16:24:02.384368 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.384326 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" event={"ID":"8575379f-09e6-4855-ae41-e804a5ee8a2f","Type":"ContainerStarted","Data":"cd8ec94b1e08af27596f67d68c03ddbb7162dfc60f68454c09362870b1c23442"} Aug 05 16:24:02.793009 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.792967 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5vc\" (UniqueName: \"kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc\") pod \"network-check-target-57h6j\" (UID: \"25e06ceb-1b89-4317-8789-90457b17cb83\") " pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:24:02.793215 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.793056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:24:02.796192 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.796164 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 16:24:02.796192 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.796183 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 05 16:24:02.806134 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.806111 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8676cf2-b5e7-47f2-895e-f5271300ea80-metrics-certs\") pod \"network-metrics-daemon-jff5g\" (UID: \"b8676cf2-b5e7-47f2-895e-f5271300ea80\") " pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:24:02.806293 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.806273 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 05 16:24:02.816474 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.816447 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x5vc\" (UniqueName: \"kubernetes.io/projected/25e06ceb-1b89-4317-8789-90457b17cb83-kube-api-access-2x5vc\") pod \"network-check-target-57h6j\" (UID: \"25e06ceb-1b89-4317-8789-90457b17cb83\") " pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:24:02.833089 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.833057 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zm8j\"" Aug 05 16:24:02.840496 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.840467 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jff5g" Aug 05 16:24:02.893825 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.893780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:24:02.897188 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.897150 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 16:24:02.907380 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:02.907353 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ddce42c-0420-4820-9d02-f1bfbdd3bf42-original-pull-secret\") pod \"global-pull-secret-syncer-c5l2j\" (UID: \"8ddce42c-0420-4820-9d02-f1bfbdd3bf42\") " pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:24:03.029449 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.029421 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t7dww\"" Aug 05 16:24:03.031572 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.031555 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5l2j" Aug 05 16:24:03.037067 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.037047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:24:03.212239 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.212206 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c5l2j"] Aug 05 16:24:03.216077 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:24:03.216048 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddce42c_0420_4820_9d02_f1bfbdd3bf42.slice/crio-add1ec1fb200906a073e764716ae652ca72c7a33eb917146a805661c0061004d WatchSource:0}: Error finding container add1ec1fb200906a073e764716ae652ca72c7a33eb917146a805661c0061004d: Status 404 returned error can't find the container with id add1ec1fb200906a073e764716ae652ca72c7a33eb917146a805661c0061004d Aug 05 16:24:03.388504 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.388430 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c5l2j" event={"ID":"8ddce42c-0420-4820-9d02-f1bfbdd3bf42","Type":"ContainerStarted","Data":"add1ec1fb200906a073e764716ae652ca72c7a33eb917146a805661c0061004d"} Aug 05 16:24:03.434428 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.434406 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jff5g"] Aug 05 16:24:03.437575 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:24:03.437536 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8676cf2_b5e7_47f2_895e_f5271300ea80.slice/crio-bdad1c2e336ef144876b8e742d4431e6af9b183991518b6ff5a8fe4a6ffdda6e WatchSource:0}: Error finding container bdad1c2e336ef144876b8e742d4431e6af9b183991518b6ff5a8fe4a6ffdda6e: Status 404 returned error can't find the container with id bdad1c2e336ef144876b8e742d4431e6af9b183991518b6ff5a8fe4a6ffdda6e Aug 05 16:24:03.437744 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.437672 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-57h6j"] Aug 05 16:24:03.441222 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:24:03.441195 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e06ceb_1b89_4317_8789_90457b17cb83.slice/crio-0a751b529f309c08c08eb8b0f464f2625397be8a8cdf7714e2bdcf93d2201b08 WatchSource:0}: Error finding container 0a751b529f309c08c08eb8b0f464f2625397be8a8cdf7714e2bdcf93d2201b08: Status 404 returned error can't find the container with id 0a751b529f309c08c08eb8b0f464f2625397be8a8cdf7714e2bdcf93d2201b08 Aug 05 16:24:03.550691 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.550663 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:03.649334 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.649240 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:03.750368 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.750337 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:03.819152 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:03.819071 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:04.395070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:04.395034 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jff5g" event={"ID":"b8676cf2-b5e7-47f2-895e-f5271300ea80","Type":"ContainerStarted","Data":"bdad1c2e336ef144876b8e742d4431e6af9b183991518b6ff5a8fe4a6ffdda6e"} Aug 05 16:24:04.397427 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:04.397401 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-57h6j" event={"ID":"25e06ceb-1b89-4317-8789-90457b17cb83","Type":"ContainerStarted","Data":"0a751b529f309c08c08eb8b0f464f2625397be8a8cdf7714e2bdcf93d2201b08"} Aug 05 16:24:04.399830 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:04.399780 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" event={"ID":"8575379f-09e6-4855-ae41-e804a5ee8a2f","Type":"ContainerStarted","Data":"1775c5cc88455840a8a0f72d17e6333aea20726968efb423fbbbe5178940c1a5"} Aug 05 16:24:04.399942 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:04.399855 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" event={"ID":"8575379f-09e6-4855-ae41-e804a5ee8a2f","Type":"ContainerStarted","Data":"be99a8332a4d468186dd8b1eaa02756e89a3193f6a47d8b90543dc86516ae58f"} Aug 05 16:24:04.420583 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:04.420542 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-gqnwh" podStartSLOduration=2.778039365 podStartE2EDuration="4.420528164s" totalImagesPullingTime="1.642488799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:04.41896559 +0000 UTC m=+67.778217436" watchObservedRunningTime="2026-08-05 16:24:04.420528164 +0000 UTC m=+67.779780007" Aug 05 16:24:05.406027 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:05.405990 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jff5g" event={"ID":"b8676cf2-b5e7-47f2-895e-f5271300ea80","Type":"ContainerStarted","Data":"3d8a8cd147fd57ba455f820c3acf3bed73f24dfcc7c1c66b6751d6cd981d9deb"} Aug 05 16:24:05.406477 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:05.406059 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jff5g" event={"ID":"b8676cf2-b5e7-47f2-895e-f5271300ea80","Type":"ContainerStarted","Data":"df9f9b712f12c03a47227ae924d170d9224aca3f8cda40fdda511c701e06532c"} Aug 05 16:24:05.422715 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:05.422430 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jff5g" podStartSLOduration=66.134813307 podStartE2EDuration="1m7.422410755s" totalImagesPullingTime="1.287597448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:05.420610556 +0000 UTC m=+68.779862404" watchObservedRunningTime="2026-08-05 16:24:05.422410755 +0000 UTC m=+68.781662598" Aug 05 16:24:05.523304 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:05.523266 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:05.583117 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:05.583078 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:05.841867 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:05.841828 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:06.102601 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.102512 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:06.129019 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.128983 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw"] Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.133077 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6qc2p"] Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.133205 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.135377 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.136579 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.137447 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.137531 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.137939 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-st59k\"" Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.138868 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.139157 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.139326 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pxz2s\"" Aug 05 16:24:06.148829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.139498 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 05 16:24:06.149750 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.149137 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw"] Aug 05 16:24:06.197943 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.197905 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:06.217933 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.217894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.218114 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.217941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-tls\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.218114 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.217977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57aad5a8-d382-423d-940e-b36b93ec0ef4-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.218114 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.217991 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22184a85-e17b-49a9-b102-58bf08245776-metrics-client-ca\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.218114 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/22184a85-e17b-49a9-b102-58bf08245776-root\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.218114 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.218114 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rf6\" (UniqueName: \"kubernetes.io/projected/22184a85-e17b-49a9-b102-58bf08245776-kube-api-access-m4rf6\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.218114 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-accelerators-collector-config\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.218114 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.218509 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-textfile\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.218509 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218141 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22184a85-e17b-49a9-b102-58bf08245776-sys\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.218509 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-wtmp\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.218509 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4448\" (UniqueName: \"kubernetes.io/projected/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-api-access-v4448\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.218509 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57aad5a8-d382-423d-940e-b36b93ec0ef4-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.218509 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.218211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.266153 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.266091 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:06.319139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4448\" (UniqueName: \"kubernetes.io/projected/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-api-access-v4448\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.319308 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57aad5a8-d382-423d-940e-b36b93ec0ef4-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.319531 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.319590 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.319644 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-tls\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.319644 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57aad5a8-d382-423d-940e-b36b93ec0ef4-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.319730 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22184a85-e17b-49a9-b102-58bf08245776-metrics-client-ca\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.319730 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319693 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/22184a85-e17b-49a9-b102-58bf08245776-root\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.319830 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.319830 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rf6\" (UniqueName: \"kubernetes.io/projected/22184a85-e17b-49a9-b102-58bf08245776-kube-api-access-m4rf6\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.319830 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319786 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-accelerators-collector-config\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.319943 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.319943 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-textfile\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.319943 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22184a85-e17b-49a9-b102-58bf08245776-sys\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.319943 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-wtmp\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.320099 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.319985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57aad5a8-d382-423d-940e-b36b93ec0ef4-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.320099 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.320077 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/22184a85-e17b-49a9-b102-58bf08245776-root\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.320099 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.320090 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-wtmp\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.322048 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:24:06.320441 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 05 16:24:06.322048 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:24:06.320512 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-tls podName:22184a85-e17b-49a9-b102-58bf08245776 nodeName:}" failed. No retries permitted until 2026-08-05 16:24:06.8204885 +0000 UTC m=+70.179740323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-tls") pod "node-exporter-6qc2p" (UID: "22184a85-e17b-49a9-b102-58bf08245776") : secret "node-exporter-tls" not found Aug 05 16:24:06.322048 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.320681 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57aad5a8-d382-423d-940e-b36b93ec0ef4-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.322048 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.321437 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.322048 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.321519 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22184a85-e17b-49a9-b102-58bf08245776-sys\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.322048 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.321549 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22184a85-e17b-49a9-b102-58bf08245776-metrics-client-ca\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.322048 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.321720 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-textfile\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.322048 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.322007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-accelerators-collector-config\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.322814 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.322755 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.324193 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.324173 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.324629 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.324607 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.328063 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.328042 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4448\" (UniqueName: \"kubernetes.io/projected/57aad5a8-d382-423d-940e-b36b93ec0ef4-kube-api-access-v4448\") pod \"kube-state-metrics-6bd9664c4-v4wtw\" (UID: \"57aad5a8-d382-423d-940e-b36b93ec0ef4\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.334672 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.332281 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rf6\" (UniqueName: \"kubernetes.io/projected/22184a85-e17b-49a9-b102-58bf08245776-kube-api-access-m4rf6\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.412682 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.412606 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:06.448132 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.448100 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" Aug 05 16:24:06.514951 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.514914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:06.769863 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.769819 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:06.824418 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.824371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-tls\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.827294 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.827266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22184a85-e17b-49a9-b102-58bf08245776-node-exporter-tls\") pod \"node-exporter-6qc2p\" (UID: \"22184a85-e17b-49a9-b102-58bf08245776\") " pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:06.873190 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:06.873152 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:07.060372 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.060291 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6qc2p" Aug 05 16:24:07.075555 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.075506 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:07.147254 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.147191 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:07.154607 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.154579 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 16:24:07.158871 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.158478 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.161829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.161504 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 05 16:24:07.161829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.161576 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gnpjl\"" Aug 05 16:24:07.161829 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.161595 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 05 16:24:07.162074 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.161841 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 05 16:24:07.162074 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.161896 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 05 16:24:07.162166 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.162105 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 05 16:24:07.162166 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.162117 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 05 16:24:07.163090 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.163069 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 05 16:24:07.163175 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.163120 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 05 16:24:07.163285 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.163270 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 05 16:24:07.172767 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.172017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 16:24:07.227849 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.227809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228037 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.227862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-out\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228037 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.227892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228037 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.227923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228037 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.227966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-volume\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228037 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.227990 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228037 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.228013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228337 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.228038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkhcq\" (UniqueName: \"kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-kube-api-access-bkhcq\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228337 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.228062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-web-config\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228337 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.228096 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228337 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.228149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228337 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.228238 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.228337 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.228277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329234 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329234 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-out\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329234 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329467 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329467 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-volume\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329467 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329467 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329467 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhcq\" (UniqueName: \"kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-kube-api-access-bkhcq\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329467 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-web-config\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329467 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329387 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329467 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329777 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329777 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329506 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.329904 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:24:07.329823 2575 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Aug 05 16:24:07.329904 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.329888 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.330020 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:24:07.329895 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-main-tls podName:9b98b3f5-a66e-476d-94d5-644cc470bddc nodeName:}" failed. No retries permitted until 2026-08-05 16:24:07.829872055 +0000 UTC m=+71.189123879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc") : secret "alertmanager-main-tls" not found Aug 05 16:24:07.330020 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:24:07.330014 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-trusted-ca-bundle podName:9b98b3f5-a66e-476d-94d5-644cc470bddc nodeName:}" failed. No retries permitted until 2026-08-05 16:24:07.829996515 +0000 UTC m=+71.189248341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc") : configmap references non-existent config key: ca-bundle.crt Aug 05 16:24:07.330134 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.330018 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.334211 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.334183 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.334412 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.334331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-volume\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.335144 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.335119 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.335416 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.335390 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-out\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.335496 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.335446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.335553 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.335541 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-web-config\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.336283 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.336254 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.337001 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.336977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.339627 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.339597 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhcq\" (UniqueName: \"kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-kube-api-access-bkhcq\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.748162 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.748123 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:07.834549 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.834505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.834769 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.834596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.835606 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.835579 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:07.837472 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:07.837444 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:08.073867 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.073761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:24:08.083978 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.083945 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-745bfdcf87-f7fxw"] Aug 05 16:24:08.086964 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.086941 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.090128 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.090099 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 05 16:24:08.090128 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.090129 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-kjjjv\"" Aug 05 16:24:08.090300 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.090153 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 05 16:24:08.090300 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.090161 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 05 16:24:08.090300 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.090101 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 05 16:24:08.090300 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.090262 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 05 16:24:08.091311 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.090101 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5ibgksdodtk7e\"" Aug 05 16:24:08.104261 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.102955 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-745bfdcf87-f7fxw"] Aug 05 16:24:08.137150 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.137114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.137150 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.137161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-grpc-tls\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.137384 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.137194 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.137384 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.137258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpqsd\" (UniqueName: \"kubernetes.io/projected/9f02957a-9c2d-4cce-a040-eb29b0fbf790-kube-api-access-zpqsd\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.137384 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.137300 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.137384 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.137347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-tls\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.137384 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.137376 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.137611 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.137406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f02957a-9c2d-4cce-a040-eb29b0fbf790-metrics-client-ca\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.238634 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.238599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.238786 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.238639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-grpc-tls\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.238786 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.238674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.238786 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.238701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpqsd\" (UniqueName: \"kubernetes.io/projected/9f02957a-9c2d-4cce-a040-eb29b0fbf790-kube-api-access-zpqsd\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.238786 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.238737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.239016 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.238784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-tls\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.239016 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.238848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.239710 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.239592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f02957a-9c2d-4cce-a040-eb29b0fbf790-metrics-client-ca\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.240674 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.240643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f02957a-9c2d-4cce-a040-eb29b0fbf790-metrics-client-ca\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.242401 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.242378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.242890 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.242813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.243272 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.243233 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.243399 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.243367 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-tls\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.244095 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.244068 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-grpc-tls\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.244233 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.244210 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f02957a-9c2d-4cce-a040-eb29b0fbf790-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.248446 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.248420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpqsd\" (UniqueName: \"kubernetes.io/projected/9f02957a-9c2d-4cce-a040-eb29b0fbf790-kube-api-access-zpqsd\") pod \"thanos-querier-745bfdcf87-f7fxw\" (UID: \"9f02957a-9c2d-4cce-a040-eb29b0fbf790\") " pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.250567 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:24:08.250539 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22184a85_e17b_49a9_b102_58bf08245776.slice/crio-a02199cf543ebd15c3ab69ab778b134175ec5eb7d9731c3b27f0838fb2434db9 WatchSource:0}: Error finding container a02199cf543ebd15c3ab69ab778b134175ec5eb7d9731c3b27f0838fb2434db9: Status 404 returned error can't find the container with id a02199cf543ebd15c3ab69ab778b134175ec5eb7d9731c3b27f0838fb2434db9 Aug 05 16:24:08.400098 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.400009 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:08.403211 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.403132 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw"] Aug 05 16:24:08.408376 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:24:08.408349 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57aad5a8_d382_423d_940e_b36b93ec0ef4.slice/crio-ee370ace93f0f323e60159de6d3bd5af124b739f808bc3e0dd22c6e095a383c3 WatchSource:0}: Error finding container ee370ace93f0f323e60159de6d3bd5af124b739f808bc3e0dd22c6e095a383c3: Status 404 returned error can't find the container with id ee370ace93f0f323e60159de6d3bd5af124b739f808bc3e0dd22c6e095a383c3 Aug 05 16:24:08.422763 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.422730 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c5l2j" event={"ID":"8ddce42c-0420-4820-9d02-f1bfbdd3bf42","Type":"ContainerStarted","Data":"8f22d931c66150d4d1ec16040d81e97ef9639276a2f544b87b225acb4e8750c2"} Aug 05 16:24:08.424017 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.423905 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-57h6j" event={"ID":"25e06ceb-1b89-4317-8789-90457b17cb83","Type":"ContainerStarted","Data":"5d44c330d9593a98b4d83e51611a921e91f5b166a1c209af06ef8e55d28342a8"} Aug 05 16:24:08.424017 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.423960 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:24:08.424745 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.424723 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qc2p" event={"ID":"22184a85-e17b-49a9-b102-58bf08245776","Type":"ContainerStarted","Data":"a02199cf543ebd15c3ab69ab778b134175ec5eb7d9731c3b27f0838fb2434db9"} Aug 05 16:24:08.425649 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.425624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" event={"ID":"57aad5a8-d382-423d-940e-b36b93ec0ef4","Type":"ContainerStarted","Data":"ee370ace93f0f323e60159de6d3bd5af124b739f808bc3e0dd22c6e095a383c3"} Aug 05 16:24:08.432307 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.432276 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 16:24:08.435474 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:24:08.435444 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b98b3f5_a66e_476d_94d5_644cc470bddc.slice/crio-aba3d6e7d08c5b62bfb5cd38305b2126fd35caef9d7b289e8f1572126df63f2e WatchSource:0}: Error finding container aba3d6e7d08c5b62bfb5cd38305b2126fd35caef9d7b289e8f1572126df63f2e: Status 404 returned error can't find the container with id aba3d6e7d08c5b62bfb5cd38305b2126fd35caef9d7b289e8f1572126df63f2e Aug 05 16:24:08.442314 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.442073 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c5l2j" podStartSLOduration=65.400619651 podStartE2EDuration="1m10.44205598s" totalImagesPullingTime="5.041436329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:08.441877178 +0000 UTC m=+71.801129023" watchObservedRunningTime="2026-08-05 16:24:08.44205598 +0000 UTC m=+71.801307820" Aug 05 16:24:08.462271 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.461716 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-57h6j" podStartSLOduration=65.657520315 podStartE2EDuration="1m10.461695067s" totalImagesPullingTime="4.804174752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:08.4604015 +0000 UTC m=+71.819653343" watchObservedRunningTime="2026-08-05 16:24:08.461695067 +0000 UTC m=+71.820946914" Aug 05 16:24:08.558755 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:08.558731 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-745bfdcf87-f7fxw"] Aug 05 16:24:08.561879 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:24:08.561844 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f02957a_9c2d_4cce_a040_eb29b0fbf790.slice/crio-9967806f9baa4e457e01d83b0a979a6e61a1bb70f297d9b8948dff3ef963ab05 WatchSource:0}: Error finding container 9967806f9baa4e457e01d83b0a979a6e61a1bb70f297d9b8948dff3ef963ab05: Status 404 returned error can't find the container with id 9967806f9baa4e457e01d83b0a979a6e61a1bb70f297d9b8948dff3ef963ab05 Aug 05 16:24:09.024705 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:09.024605 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:09.075092 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:09.075020 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:09.182590 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:09.182518 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:09.424004 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:09.423822 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:09.432608 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:09.432494 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" event={"ID":"9f02957a-9c2d-4cce-a040-eb29b0fbf790","Type":"ContainerStarted","Data":"9967806f9baa4e457e01d83b0a979a6e61a1bb70f297d9b8948dff3ef963ab05"} Aug 05 16:24:09.435600 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:09.435476 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerStarted","Data":"aba3d6e7d08c5b62bfb5cd38305b2126fd35caef9d7b289e8f1572126df63f2e"} Aug 05 16:24:10.468505 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:10.468460 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:10.533867 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:10.533831 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:10.869884 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:10.869627 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:10.996143 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:10.996111 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:11.447678 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.447589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" event={"ID":"57aad5a8-d382-423d-940e-b36b93ec0ef4","Type":"ContainerStarted","Data":"f5f3a6873a7231dd3deb7305981422b4d1bbe93a5ce343259e745668e1572fc1"} Aug 05 16:24:11.447678 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.447659 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" event={"ID":"57aad5a8-d382-423d-940e-b36b93ec0ef4","Type":"ContainerStarted","Data":"86f23c3c97c61be9e37179c36c4604456577cf25dd39da507f262eb8d9567297"} Aug 05 16:24:11.447923 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.447688 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" event={"ID":"57aad5a8-d382-423d-940e-b36b93ec0ef4","Type":"ContainerStarted","Data":"d4a84e235ab97bd80fb9a239e2ff3f7e8df2995ad12d357a08e681658b5aaffd"} Aug 05 16:24:11.449445 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.449417 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" event={"ID":"9f02957a-9c2d-4cce-a040-eb29b0fbf790","Type":"ContainerStarted","Data":"9c6d28ff27732ce5649ca3aedc64d2734296aabae4cc4a6ab3b79030212c8f2c"} Aug 05 16:24:11.449556 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.449472 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" event={"ID":"9f02957a-9c2d-4cce-a040-eb29b0fbf790","Type":"ContainerStarted","Data":"acd4d99e6feca2abcbcd5c101296a34ec625be0457028f367995b53facb0c465"} Aug 05 16:24:11.449556 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.449498 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" event={"ID":"9f02957a-9c2d-4cce-a040-eb29b0fbf790","Type":"ContainerStarted","Data":"6ca6696253b8de77f28c0f219f9b0e9cb67157d7cdc6194a59feb2e4c88ab9ba"} Aug 05 16:24:11.450656 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.450633 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerID="a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40" exitCode=0 Aug 05 16:24:11.450739 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.450718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerDied","Data":"a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40"} Aug 05 16:24:11.452154 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.452126 2575 generic.go:356] "Generic (PLEG): container finished" podID="22184a85-e17b-49a9-b102-58bf08245776" containerID="ad3e16431b829f02e29fc5fdd7b58ead41c1269587cf312a6494c068ad67d284" exitCode=0 Aug 05 16:24:11.452234 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.452167 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qc2p" event={"ID":"22184a85-e17b-49a9-b102-58bf08245776","Type":"ContainerDied","Data":"ad3e16431b829f02e29fc5fdd7b58ead41c1269587cf312a6494c068ad67d284"} Aug 05 16:24:11.469981 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:11.469940 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-v4wtw" podStartSLOduration=2.955542474 podStartE2EDuration="5.469926706s" totalImagesPullingTime="2.514384232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:11.468462187 +0000 UTC m=+74.827714029" watchObservedRunningTime="2026-08-05 16:24:11.469926706 +0000 UTC m=+74.829178572" Aug 05 16:24:12.294662 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:12.294631 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:12.461583 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:12.461483 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" event={"ID":"9f02957a-9c2d-4cce-a040-eb29b0fbf790","Type":"ContainerStarted","Data":"65b43520fbe0e833875452ffc69a3b2492489207152b7d5a38a6f02865839b75"} Aug 05 16:24:12.461583 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:12.461568 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" event={"ID":"9f02957a-9c2d-4cce-a040-eb29b0fbf790","Type":"ContainerStarted","Data":"9c3c64c5073f46b12e5cd2210faca149250416f87b2a9abaee8ea7188bebfbd1"} Aug 05 16:24:12.461900 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:12.461616 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" event={"ID":"9f02957a-9c2d-4cce-a040-eb29b0fbf790","Type":"ContainerStarted","Data":"e9c6a8fe5aed132e7123745cd878354e3c0862baebd24facf1a66daed986b78e"} Aug 05 16:24:12.461900 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:12.461660 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:12.465355 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:12.464778 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qc2p" event={"ID":"22184a85-e17b-49a9-b102-58bf08245776","Type":"ContainerStarted","Data":"c75967cc492ad458c2d09cfead61ae0cd2bd50ee97334ffda421ebff9d485aff"} Aug 05 16:24:12.465355 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:12.464859 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qc2p" event={"ID":"22184a85-e17b-49a9-b102-58bf08245776","Type":"ContainerStarted","Data":"4187d6f120dc4cd285bff0f3d0dbfc015d78d02c44abb02a9ff6f9adb92507f8"} Aug 05 16:24:12.487902 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:12.487722 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" podStartSLOduration=1.3235164369999999 podStartE2EDuration="4.487699987s" totalImagesPullingTime="3.16418355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:24:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:12.484541006 +0000 UTC m=+75.843792847" watchObservedRunningTime="2026-08-05 16:24:12.487699987 +0000 UTC m=+75.846951832" Aug 05 16:24:12.506245 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:12.504256 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6qc2p" podStartSLOduration=3.790715905 podStartE2EDuration="6.504234884s" totalImagesPullingTime="2.616694513s" totalInitContainerRuntime="96.824466ms" isStatefulPod=true podCreationTimestamp="2026-08-05 16:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:12.502971311 +0000 UTC m=+75.862223156" watchObservedRunningTime="2026-08-05 16:24:12.504234884 +0000 UTC m=+75.863486729" Aug 05 16:24:13.472144 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:13.472107 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerStarted","Data":"2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2"} Aug 05 16:24:13.472317 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:13.472195 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerStarted","Data":"6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6"} Aug 05 16:24:13.472317 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:13.472233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerStarted","Data":"68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b"} Aug 05 16:24:13.472317 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:13.472265 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerStarted","Data":"b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5"} Aug 05 16:24:13.472317 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:13.472300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerStarted","Data":"1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d"} Aug 05 16:24:13.472532 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:13.472335 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerStarted","Data":"dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c"} Aug 05 16:24:13.506611 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:13.506560 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.382383173 podStartE2EDuration="6.506546524s" totalImagesPullingTime="4.094637561s" totalInitContainerRuntime="29.52579ms" isStatefulPod=false podCreationTimestamp="2026-08-05 16:24:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:13.504084988 +0000 UTC m=+76.863336834" watchObservedRunningTime="2026-08-05 16:24:13.506546524 +0000 UTC m=+76.865798366" Aug 05 16:24:14.359831 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:14.359785 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6ff4748cdb-cslpc" Aug 05 16:24:14.509021 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:14.508986 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:14.600760 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:14.600731 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:14.700946 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:14.700854 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:18.478475 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:18.478363 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-745bfdcf87-f7fxw" Aug 05 16:24:19.001595 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.001562 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-wv2fb"] Aug 05 16:24:19.004836 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.004821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-wv2fb" Aug 05 16:24:19.007895 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.007875 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jv7wv\"" Aug 05 16:24:19.008230 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.008214 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 05 16:24:19.008737 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.008724 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 05 16:24:19.018273 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.018253 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-wv2fb"] Aug 05 16:24:19.022577 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.022558 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:19.121397 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.121362 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:19.132846 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.132824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brgcj\" (UniqueName: \"kubernetes.io/projected/7e9325ac-4eaa-42c5-8bd1-5e35a32d6629-kube-api-access-brgcj\") pod \"downloads-77c974ffcf-wv2fb\" (UID: \"7e9325ac-4eaa-42c5-8bd1-5e35a32d6629\") " pod="openshift-console/downloads-77c974ffcf-wv2fb" Aug 05 16:24:19.194154 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.194126 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:19.233234 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.233205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brgcj\" (UniqueName: \"kubernetes.io/projected/7e9325ac-4eaa-42c5-8bd1-5e35a32d6629-kube-api-access-brgcj\") pod \"downloads-77c974ffcf-wv2fb\" (UID: \"7e9325ac-4eaa-42c5-8bd1-5e35a32d6629\") " pod="openshift-console/downloads-77c974ffcf-wv2fb" Aug 05 16:24:19.243112 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.243077 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brgcj\" (UniqueName: \"kubernetes.io/projected/7e9325ac-4eaa-42c5-8bd1-5e35a32d6629-kube-api-access-brgcj\") pod \"downloads-77c974ffcf-wv2fb\" (UID: \"7e9325ac-4eaa-42c5-8bd1-5e35a32d6629\") " pod="openshift-console/downloads-77c974ffcf-wv2fb" Aug 05 16:24:19.314363 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.314285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-wv2fb" Aug 05 16:24:19.470337 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.470266 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-wv2fb"] Aug 05 16:24:19.473558 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:24:19.473531 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9325ac_4eaa_42c5_8bd1_5e35a32d6629.slice/crio-a5ee5a5fc20d369c38418a27a764583495bc64c283dcde7add7e717a89dd9391 WatchSource:0}: Error finding container a5ee5a5fc20d369c38418a27a764583495bc64c283dcde7add7e717a89dd9391: Status 404 returned error can't find the container with id a5ee5a5fc20d369c38418a27a764583495bc64c283dcde7add7e717a89dd9391 Aug 05 16:24:19.499221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.499188 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-wv2fb" event={"ID":"7e9325ac-4eaa-42c5-8bd1-5e35a32d6629","Type":"ContainerStarted","Data":"a5ee5a5fc20d369c38418a27a764583495bc64c283dcde7add7e717a89dd9391"} Aug 05 16:24:19.547291 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.547252 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:19.674422 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.674341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:19.774881 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:19.774850 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:20.069509 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:20.069481 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:20.124582 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:20.124551 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:20.273014 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:20.272959 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:20.372411 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:20.372337 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:20.919218 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:20.919187 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:21.086376 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:21.086346 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:21.316586 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:21.316559 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:21.543731 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:21.543703 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:21.601564 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:21.601462 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:21.899820 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:21.899699 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:21.964216 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:21.964186 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:23.937321 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:23.937282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:23.995374 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:23.995342 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:24.902636 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:24.902599 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:25.002509 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:25.002468 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:25.062660 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:25.062623 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:26.124692 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:26.124651 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:26.225899 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:26.225866 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:30.557196 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:30.557164 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:30.657930 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:30.657891 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:33.021217 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.021187 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:33.077282 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.077247 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:33.106214 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.106182 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-657cc76667-j5b2f"] Aug 05 16:24:33.131166 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.131135 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-657cc76667-j5b2f"] Aug 05 16:24:33.131347 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.131259 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.134477 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.134456 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6kwgg\"" Aug 05 16:24:33.134626 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.134521 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 05 16:24:33.134626 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.134580 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 05 16:24:33.134844 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.134827 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 05 16:24:33.135310 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.134862 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 05 16:24:33.136078 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.136058 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 05 16:24:33.145249 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.145228 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 05 16:24:33.193471 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.193441 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:33.255009 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.254974 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:33.261177 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.261148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-trusted-ca-bundle\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.261307 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.261196 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-config\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.261307 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.261215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-service-ca\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.261412 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.261326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrqn2\" (UniqueName: \"kubernetes.io/projected/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-kube-api-access-rrqn2\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.261412 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.261384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-oauth-config\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.261489 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.261426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-serving-cert\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.261489 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.261449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-oauth-serving-cert\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.362197 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.362127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-trusted-ca-bundle\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.362197 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.362177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-config\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.362427 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.362205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-service-ca\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.362427 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.362252 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrqn2\" (UniqueName: \"kubernetes.io/projected/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-kube-api-access-rrqn2\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.362427 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.362272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-oauth-config\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.362427 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.362324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-serving-cert\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.362427 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.362353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-oauth-serving-cert\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.362954 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.362932 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-config\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.363141 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.363117 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-oauth-serving-cert\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.363217 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.363136 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-service-ca\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.363217 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.363162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-trusted-ca-bundle\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.365187 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.365166 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-serving-cert\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.365275 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.365223 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-oauth-config\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.371195 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.371171 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrqn2\" (UniqueName: \"kubernetes.io/projected/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-kube-api-access-rrqn2\") pod \"console-657cc76667-j5b2f\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.447021 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.446966 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:33.709593 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.709489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:33.810503 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:33.810464 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:34.686418 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:34.686382 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:34.786973 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:34.786937 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:35.491097 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:35.491067 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-657cc76667-j5b2f"] Aug 05 16:24:35.503694 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:24:35.503668 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cd3ff5_2ba8_4f53_aded_dd5aa768aa32.slice/crio-e25e0c5a37a50b791f6835278a480ed6e7f553bc4d710ac9344116270a417ee2 WatchSource:0}: Error finding container e25e0c5a37a50b791f6835278a480ed6e7f553bc4d710ac9344116270a417ee2: Status 404 returned error can't find the container with id e25e0c5a37a50b791f6835278a480ed6e7f553bc4d710ac9344116270a417ee2 Aug 05 16:24:35.569151 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:35.569124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657cc76667-j5b2f" event={"ID":"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32","Type":"ContainerStarted","Data":"e25e0c5a37a50b791f6835278a480ed6e7f553bc4d710ac9344116270a417ee2"} Aug 05 16:24:36.576032 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:36.575995 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-wv2fb" event={"ID":"7e9325ac-4eaa-42c5-8bd1-5e35a32d6629","Type":"ContainerStarted","Data":"7c0161420df8ab877b4e44b521440a32dfb93e706018c59c5139199b752b3429"} Aug 05 16:24:36.576957 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:36.576551 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-wv2fb" Aug 05 16:24:36.592094 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:36.592069 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-wv2fb" Aug 05 16:24:36.615923 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:36.615869 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-wv2fb" podStartSLOduration=2.5354473520000003 podStartE2EDuration="18.615847504s" totalImagesPullingTime="16.080400152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:36.59800335 +0000 UTC m=+99.957255195" watchObservedRunningTime="2026-08-05 16:24:36.615847504 +0000 UTC m=+99.975099354" Aug 05 16:24:37.100187 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:37.100151 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:37.158281 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:37.158233 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:39.438628 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:39.438598 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-57h6j" Aug 05 16:24:40.596622 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:40.596585 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657cc76667-j5b2f" event={"ID":"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32","Type":"ContainerStarted","Data":"23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0"} Aug 05 16:24:40.616481 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:40.616426 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-657cc76667-j5b2f" podStartSLOduration=3.26856293 podStartE2EDuration="7.616410279s" totalImagesPullingTime="4.347847349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:40.613965254 +0000 UTC m=+103.973217135" watchObservedRunningTime="2026-08-05 16:24:40.616410279 +0000 UTC m=+103.975662121" Aug 05 16:24:43.447534 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:43.447491 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:43.448044 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:43.447563 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:43.453359 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:43.453330 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:43.615783 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:43.615748 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:24:44.389953 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:44.389918 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:44.920087 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:44.920042 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:45.021536 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:45.021496 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:45.703326 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:45.703276 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:45.803484 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:45.803457 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:48.193377 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:48.193346 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:48.252823 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:24:48.252777 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:03.723028 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:03.722991 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:25.334174 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:25.334138 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:25.721731 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:25.721653 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:26.380394 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.380365 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 16:25:26.380817 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.380728 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="alertmanager" containerID="cri-o://dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c" gracePeriod=120 Aug 05 16:25:26.380817 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.380772 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy-metric" containerID="cri-o://6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6" gracePeriod=120 Aug 05 16:25:26.380967 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.380832 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="prom-label-proxy" containerID="cri-o://2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2" gracePeriod=120 Aug 05 16:25:26.381025 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.380852 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy-web" containerID="cri-o://b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5" gracePeriod=120 Aug 05 16:25:26.381025 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.380855 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="config-reloader" containerID="cri-o://1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d" gracePeriod=120 Aug 05 16:25:26.381116 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.381015 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy" containerID="cri-o://68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b" gracePeriod=120 Aug 05 16:25:26.798492 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.798459 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerID="2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2" exitCode=0 Aug 05 16:25:26.798492 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.798483 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerID="68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b" exitCode=0 Aug 05 16:25:26.798492 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.798489 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerID="1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d" exitCode=0 Aug 05 16:25:26.798492 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.798495 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerID="dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c" exitCode=0 Aug 05 16:25:26.798746 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.798525 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerDied","Data":"2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2"} Aug 05 16:25:26.798746 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.798612 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerDied","Data":"68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b"} Aug 05 16:25:26.798746 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.798637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerDied","Data":"1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d"} Aug 05 16:25:26.798746 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:26.798660 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerDied","Data":"dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c"} Aug 05 16:25:27.625306 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.625283 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:27.702982 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.702915 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.702982 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.702950 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-web\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.702982 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.702976 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkhcq\" (UniqueName: \"kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-kube-api-access-bkhcq\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.703226 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703004 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-out\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.703226 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703051 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-cluster-tls-config\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.703226 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703086 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-volume\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.703226 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703116 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-trusted-ca-bundle\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.703226 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703150 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-main-db\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.703226 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703180 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-main-tls\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.703226 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703208 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.703531 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703243 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-web-config\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.703531 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703270 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-tls-assets\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.703531 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703321 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-metrics-client-ca\") pod \"9b98b3f5-a66e-476d-94d5-644cc470bddc\" (UID: \"9b98b3f5-a66e-476d-94d5-644cc470bddc\") " Aug 05 16:25:27.704317 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.703845 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:25:27.704317 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.704054 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:25:27.704504 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.704399 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:25:27.707623 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.707496 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-kube-api-access-bkhcq" (OuterVolumeSpecName: "kube-api-access-bkhcq") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "kube-api-access-bkhcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:25:27.707623 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.707422 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-volume" (OuterVolumeSpecName: "config-volume") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:25:27.707815 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.707721 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-out" (OuterVolumeSpecName: "config-out") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:25:27.707815 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.707762 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:25:27.707969 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.707847 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:25:27.707969 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.707872 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:25:27.707969 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.707939 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:25:27.708180 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.708164 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:25:27.713438 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.713412 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:25:27.718221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.718199 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-web-config" (OuterVolumeSpecName: "web-config") pod "9b98b3f5-a66e-476d-94d5-644cc470bddc" (UID: "9b98b3f5-a66e-476d-94d5-644cc470bddc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:25:27.803873 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803850 2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-cluster-tls-config\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.803873 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803868 2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-volume\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.803873 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803878 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.804024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803886 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-alertmanager-main-db\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.804024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803894 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-main-tls\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.804024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803905 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.804024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803935 2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-web-config\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.804024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803943 2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-tls-assets\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.804024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803950 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b98b3f5-a66e-476d-94d5-644cc470bddc-metrics-client-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.804024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803958 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.804024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803975 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b98b3f5-a66e-476d-94d5-644cc470bddc-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.804024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803985 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkhcq\" (UniqueName: \"kubernetes.io/projected/9b98b3f5-a66e-476d-94d5-644cc470bddc-kube-api-access-bkhcq\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.804024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.803994 2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b98b3f5-a66e-476d-94d5-644cc470bddc-config-out\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:25:27.805584 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.805551 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerID="6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6" exitCode=0 Aug 05 16:25:27.805584 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.805585 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerID="b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5" exitCode=0 Aug 05 16:25:27.805732 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.805640 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerDied","Data":"6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6"} Aug 05 16:25:27.805732 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.805657 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:27.805732 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.805720 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerDied","Data":"b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5"} Aug 05 16:25:27.805914 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.805750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b98b3f5-a66e-476d-94d5-644cc470bddc","Type":"ContainerDied","Data":"aba3d6e7d08c5b62bfb5cd38305b2126fd35caef9d7b289e8f1572126df63f2e"} Aug 05 16:25:27.805914 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.805781 2575 scope.go:117] "RemoveContainer" containerID="2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2" Aug 05 16:25:27.816475 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.816455 2575 scope.go:117] "RemoveContainer" containerID="6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6" Aug 05 16:25:27.823298 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.823284 2575 scope.go:117] "RemoveContainer" containerID="68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b" Aug 05 16:25:27.830700 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.830660 2575 scope.go:117] "RemoveContainer" containerID="b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5" Aug 05 16:25:27.831215 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.831196 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 16:25:27.837224 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.837206 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 16:25:27.840172 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.840157 2575 scope.go:117] "RemoveContainer" containerID="1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d" Aug 05 16:25:27.846915 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.846900 2575 scope.go:117] "RemoveContainer" containerID="dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c" Aug 05 16:25:27.853407 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.853389 2575 scope.go:117] "RemoveContainer" containerID="a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40" Aug 05 16:25:27.859963 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.859947 2575 scope.go:117] "RemoveContainer" containerID="2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2" Aug 05 16:25:27.860203 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:25:27.860184 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2\": container with ID starting with 2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2 not found: ID does not exist" containerID="2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2" Aug 05 16:25:27.860246 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.860211 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2"} err="failed to get container status \"2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2\": rpc error: code = NotFound desc = could not find container \"2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2\": container with ID starting with 2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2 not found: ID does not exist" Aug 05 16:25:27.860246 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.860240 2575 scope.go:117] "RemoveContainer" containerID="6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6" Aug 05 16:25:27.860459 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:25:27.860441 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6\": container with ID starting with 6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6 not found: ID does not exist" containerID="6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6" Aug 05 16:25:27.860501 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.860464 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6"} err="failed to get container status \"6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6\": rpc error: code = NotFound desc = could not find container \"6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6\": container with ID starting with 6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6 not found: ID does not exist" Aug 05 16:25:27.860501 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.860479 2575 scope.go:117] "RemoveContainer" containerID="68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b" Aug 05 16:25:27.860729 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:25:27.860712 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b\": container with ID starting with 68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b not found: ID does not exist" containerID="68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b" Aug 05 16:25:27.860844 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.860759 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b"} err="failed to get container status \"68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b\": rpc error: code = NotFound desc = could not find container \"68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b\": container with ID starting with 68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b not found: ID does not exist" Aug 05 16:25:27.860844 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.860783 2575 scope.go:117] "RemoveContainer" containerID="b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5" Aug 05 16:25:27.861071 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:25:27.861055 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5\": container with ID starting with b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5 not found: ID does not exist" containerID="b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5" Aug 05 16:25:27.861115 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.861074 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5"} err="failed to get container status \"b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5\": rpc error: code = NotFound desc = could not find container \"b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5\": container with ID starting with b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5 not found: ID does not exist" Aug 05 16:25:27.861115 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.861087 2575 scope.go:117] "RemoveContainer" containerID="1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d" Aug 05 16:25:27.861258 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:25:27.861244 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d\": container with ID starting with 1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d not found: ID does not exist" containerID="1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d" Aug 05 16:25:27.861298 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.861262 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d"} err="failed to get container status \"1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d\": rpc error: code = NotFound desc = could not find container \"1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d\": container with ID starting with 1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d not found: ID does not exist" Aug 05 16:25:27.861298 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.861276 2575 scope.go:117] "RemoveContainer" containerID="dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c" Aug 05 16:25:27.861460 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:25:27.861446 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c\": container with ID starting with dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c not found: ID does not exist" containerID="dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c" Aug 05 16:25:27.861501 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.861465 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c"} err="failed to get container status \"dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c\": rpc error: code = NotFound desc = could not find container \"dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c\": container with ID starting with dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c not found: ID does not exist" Aug 05 16:25:27.861501 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.861479 2575 scope.go:117] "RemoveContainer" containerID="a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40" Aug 05 16:25:27.861647 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:25:27.861630 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40\": container with ID starting with a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40 not found: ID does not exist" containerID="a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40" Aug 05 16:25:27.861683 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.861652 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40"} err="failed to get container status \"a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40\": rpc error: code = NotFound desc = could not find container \"a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40\": container with ID starting with a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40 not found: ID does not exist" Aug 05 16:25:27.861683 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.861664 2575 scope.go:117] "RemoveContainer" containerID="2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2" Aug 05 16:25:27.861849 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.861834 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2"} err="failed to get container status \"2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2\": rpc error: code = NotFound desc = could not find container \"2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2\": container with ID starting with 2cc727a3a8f033059ca1228b8eb9f669bd76b7f034d86b8bfb812687d6d52bd2 not found: ID does not exist" Aug 05 16:25:27.861911 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.861849 2575 scope.go:117] "RemoveContainer" containerID="6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6" Aug 05 16:25:27.862035 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862020 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6"} err="failed to get container status \"6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6\": rpc error: code = NotFound desc = could not find container \"6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6\": container with ID starting with 6ad334a101c3c62830974446caa846ce858a04bc78544af8b9004249e73cb9c6 not found: ID does not exist" Aug 05 16:25:27.862075 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862037 2575 scope.go:117] "RemoveContainer" containerID="68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b" Aug 05 16:25:27.862186 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862171 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b"} err="failed to get container status \"68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b\": rpc error: code = NotFound desc = could not find container \"68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b\": container with ID starting with 68b35130a46ed35e7f14f5a6ccd1c855dc0320c8292469eef154d6be5f51bb0b not found: ID does not exist" Aug 05 16:25:27.862230 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862186 2575 scope.go:117] "RemoveContainer" containerID="b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5" Aug 05 16:25:27.862373 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862358 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5"} err="failed to get container status \"b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5\": rpc error: code = NotFound desc = could not find container \"b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5\": container with ID starting with b7835dd63e0798abcb0c3722200d3dd7b10767a844febf1b2b7386de337a17f5 not found: ID does not exist" Aug 05 16:25:27.862414 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862374 2575 scope.go:117] "RemoveContainer" containerID="1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d" Aug 05 16:25:27.862585 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862563 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d"} err="failed to get container status \"1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d\": rpc error: code = NotFound desc = could not find container \"1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d\": container with ID starting with 1189039ba779d435b350bbcf5c0595de95b09c3ed87c8c20cc58e36f7177cc9d not found: ID does not exist" Aug 05 16:25:27.862644 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862585 2575 scope.go:117] "RemoveContainer" containerID="dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c" Aug 05 16:25:27.862786 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862770 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c"} err="failed to get container status \"dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c\": rpc error: code = NotFound desc = could not find container \"dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c\": container with ID starting with dc5172ba22e96e460c4cf648e5c1240acc18403dcef4c04784f2cac78a2a366c not found: ID does not exist" Aug 05 16:25:27.862855 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862786 2575 scope.go:117] "RemoveContainer" containerID="a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40" Aug 05 16:25:27.862991 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.862976 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40"} err="failed to get container status \"a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40\": rpc error: code = NotFound desc = could not find container \"a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40\": container with ID starting with a2c427104c188865ee9d3fd7a64a1a093bc1888bac13038aa33ce05fd9253a40 not found: ID does not exist" Aug 05 16:25:27.867414 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867389 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 16:25:27.867653 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867643 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy-web" Aug 05 16:25:27.867696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867655 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy-web" Aug 05 16:25:27.867696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867666 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="init-config-reloader" Aug 05 16:25:27.867696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867672 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="init-config-reloader" Aug 05 16:25:27.867696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867682 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy" Aug 05 16:25:27.867696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867688 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy" Aug 05 16:25:27.867696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867696 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="config-reloader" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867701 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="config-reloader" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867707 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="alertmanager" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867712 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="alertmanager" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867721 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy-metric" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867726 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy-metric" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867730 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="prom-label-proxy" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867735 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="prom-label-proxy" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867778 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="alertmanager" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867784 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="config-reloader" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867813 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy-web" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867819 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy-metric" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867824 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="kube-rbac-proxy" Aug 05 16:25:27.867885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.867829 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" containerName="prom-label-proxy" Aug 05 16:25:27.871427 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.871409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:27.874242 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.874222 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 05 16:25:27.874333 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.874231 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 05 16:25:27.874493 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.874479 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 05 16:25:27.874563 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.874536 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 05 16:25:27.874563 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.874539 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gnpjl\"" Aug 05 16:25:27.874658 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.874536 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 05 16:25:27.874658 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.874584 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 05 16:25:27.874747 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.874677 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 05 16:25:27.874747 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.874712 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 05 16:25:27.881070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.881054 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 05 16:25:27.893145 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:27.893123 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 16:25:28.004985 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.004955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005116 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.004992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005116 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr2cc\" (UniqueName: \"kubernetes.io/projected/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-kube-api-access-tr2cc\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005116 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005035 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005229 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005229 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005171 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-web-config\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005229 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005201 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005309 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005309 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005369 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-config-out\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005409 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005488 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.005523 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.005501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.105981 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.105938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.105981 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.105979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.105997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr2cc\" (UniqueName: \"kubernetes.io/projected/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-kube-api-access-tr2cc\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.106021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.106041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.106073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-web-config\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.106188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.106231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106519 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.106283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106519 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.106313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-config-out\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106519 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.106370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106519 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.106427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.106519 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.106460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.107252 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.107225 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.108463 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.107366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.108463 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.108131 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.109490 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.109158 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.109490 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.109249 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.109490 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.109418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-web-config\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.109490 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.109461 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.109734 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.109517 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.109734 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.109676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.109817 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.109768 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-config-out\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.110696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.110676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.111246 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.111229 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.115240 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.115222 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr2cc\" (UniqueName: \"kubernetes.io/projected/7ef73249-a007-4cfb-b75c-a5c4e65ff9b4-kube-api-access-tr2cc\") pod \"alertmanager-main-0\" (UID: \"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.180523 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.180480 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 16:25:28.320535 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.320507 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 16:25:28.322600 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:25:28.322574 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef73249_a007_4cfb_b75c_a5c4e65ff9b4.slice/crio-a2e6b264913cbb42ef97278ddd07fc3525cf2ff44e01f9ead72186524ddb1892 WatchSource:0}: Error finding container a2e6b264913cbb42ef97278ddd07fc3525cf2ff44e01f9ead72186524ddb1892: Status 404 returned error can't find the container with id a2e6b264913cbb42ef97278ddd07fc3525cf2ff44e01f9ead72186524ddb1892 Aug 05 16:25:28.811902 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.811871 2575 generic.go:356] "Generic (PLEG): container finished" podID="7ef73249-a007-4cfb-b75c-a5c4e65ff9b4" containerID="472b33ea6dd089802c52caf5dbfdc1325f9aea484f44f46adae64e101a9e42c3" exitCode=0 Aug 05 16:25:28.812252 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.811936 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4","Type":"ContainerDied","Data":"472b33ea6dd089802c52caf5dbfdc1325f9aea484f44f46adae64e101a9e42c3"} Aug 05 16:25:28.812252 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:28.811989 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4","Type":"ContainerStarted","Data":"a2e6b264913cbb42ef97278ddd07fc3525cf2ff44e01f9ead72186524ddb1892"} Aug 05 16:25:29.120906 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:29.120878 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b98b3f5-a66e-476d-94d5-644cc470bddc" path="/var/lib/kubelet/pods/9b98b3f5-a66e-476d-94d5-644cc470bddc/volumes" Aug 05 16:25:29.818404 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:29.818368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4","Type":"ContainerStarted","Data":"0a3c9ebd10c902c6482bda00380ef6f2930645b2ef0ea935fdfc4f8d98b28655"} Aug 05 16:25:29.818811 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:29.818429 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4","Type":"ContainerStarted","Data":"88e4713b7e410b91c3ed732ae66c464abc1dfab3714d8672fe4f4677a8dc84d3"} Aug 05 16:25:29.818811 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:29.818454 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4","Type":"ContainerStarted","Data":"1194b72f912eb21026b31e9386d6b1f44e34c40063e05f91ae69b71eddf2524c"} Aug 05 16:25:29.818811 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:29.818482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4","Type":"ContainerStarted","Data":"54664df431a798ed5a02cf04346b7e8d4a9922b3f59c2c4f97409f6efdb58c84"} Aug 05 16:25:29.818811 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:29.818506 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4","Type":"ContainerStarted","Data":"108eec3d9a8c38ecebaf5d3d01b1d8293bf8a103b988b7c7602ff2a1a0d41059"} Aug 05 16:25:29.818811 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:29.818527 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ef73249-a007-4cfb-b75c-a5c4e65ff9b4","Type":"ContainerStarted","Data":"b95b81391d3ac9e5ab4658fe0cefcd6f90a38001fb884b30c2c083dad13ff416"} Aug 05 16:25:29.869717 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:29.869669 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.85435993 podStartE2EDuration="2.86965457s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.29464ms" isStatefulPod=false podCreationTimestamp="2026-08-05 16:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:25:29.868490203 +0000 UTC m=+153.227742039" watchObservedRunningTime="2026-08-05 16:25:29.86965457 +0000 UTC m=+153.228906415" Aug 05 16:25:30.415893 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.415854 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl"] Aug 05 16:25:30.424544 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.424512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.427600 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.427574 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 05 16:25:30.427729 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.427575 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 05 16:25:30.427827 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.427747 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 05 16:25:30.427827 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.427788 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 05 16:25:30.427827 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.427816 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5tgv6\"" Aug 05 16:25:30.427994 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.427747 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 05 16:25:30.433702 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.433637 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 05 16:25:30.435197 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.435148 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl"] Aug 05 16:25:30.525642 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.525607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-secret-telemeter-client\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.525826 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.525650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-federate-client-tls\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.525826 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.525678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6lvb\" (UniqueName: \"kubernetes.io/projected/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-kube-api-access-h6lvb\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.525826 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.525768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.525962 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.525834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-serving-certs-ca-bundle\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.525962 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.525923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-telemeter-client-tls\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.525962 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.525950 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-metrics-client-ca\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.526088 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.525973 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.627333 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.627293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-serving-certs-ca-bundle\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.627508 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.627366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-telemeter-client-tls\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.627508 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.627399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-metrics-client-ca\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.627508 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.627422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.627508 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.627500 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-secret-telemeter-client\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.627714 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.627524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-federate-client-tls\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.627714 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.627689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6lvb\" (UniqueName: \"kubernetes.io/projected/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-kube-api-access-h6lvb\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.627830 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.627751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.628228 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.628124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-serving-certs-ca-bundle\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.628391 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.628366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-metrics-client-ca\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.628463 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.628428 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.630539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.630495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-telemeter-client-tls\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.630852 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.630813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-federate-client-tls\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.630948 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.630914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-secret-telemeter-client\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.631882 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.631470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.639589 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.639561 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6lvb\" (UniqueName: \"kubernetes.io/projected/65f07a07-c83c-4433-bf9a-64dd7cd5f2a6-kube-api-access-h6lvb\") pod \"telemeter-client-6b7fc9bb58-s7gcl\" (UID: \"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6\") " pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.738293 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.738219 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" Aug 05 16:25:30.887072 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:30.887041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl"] Aug 05 16:25:30.892819 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:25:30.892777 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f07a07_c83c_4433_bf9a_64dd7cd5f2a6.slice/crio-52608325dc36c070b8b654a8d40cbf8c9d80d95ebbbf7782fc3e65528ded09de WatchSource:0}: Error finding container 52608325dc36c070b8b654a8d40cbf8c9d80d95ebbbf7782fc3e65528ded09de: Status 404 returned error can't find the container with id 52608325dc36c070b8b654a8d40cbf8c9d80d95ebbbf7782fc3e65528ded09de Aug 05 16:25:31.830545 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:31.830512 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" event={"ID":"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6","Type":"ContainerStarted","Data":"52608325dc36c070b8b654a8d40cbf8c9d80d95ebbbf7782fc3e65528ded09de"} Aug 05 16:25:33.844041 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:33.844006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" event={"ID":"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6","Type":"ContainerStarted","Data":"32a21825c8b1f20690bbd81e8ba3f41bc22e81b953a1416422e21db8df28f0c6"} Aug 05 16:25:33.844438 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:33.844057 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" event={"ID":"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6","Type":"ContainerStarted","Data":"347263bbf01676c8867eb9ef677336fba9a158b03d359e057fdb34b50bba9912"} Aug 05 16:25:33.844438 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:33.844076 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" event={"ID":"65f07a07-c83c-4433-bf9a-64dd7cd5f2a6","Type":"ContainerStarted","Data":"fe695c29f525572becdbf3d3234b5af430e22eeb8ed71031cad1a7d661c36f08"} Aug 05 16:25:33.868398 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:33.868350 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6b7fc9bb58-s7gcl" podStartSLOduration=1.8345019439999999 podStartE2EDuration="3.868338838s" totalImagesPullingTime="2.033836894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:25:33.866783859 +0000 UTC m=+157.226035714" watchObservedRunningTime="2026-08-05 16:25:33.868338838 +0000 UTC m=+157.227590681" Aug 05 16:25:33.959978 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:33.959954 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-556ffd758f-8tzln"] Aug 05 16:25:33.962469 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:33.962454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:33.975988 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:33.975966 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556ffd758f-8tzln"] Aug 05 16:25:34.022370 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.022330 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:34.057382 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.057358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-config\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.057476 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.057398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-oauth-serving-cert\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.057476 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.057421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-serving-cert\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.057555 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.057486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-oauth-config\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.057555 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.057530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-service-ca\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.057555 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.057547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45fsx\" (UniqueName: \"kubernetes.io/projected/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-kube-api-access-45fsx\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.057667 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.057568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-trusted-ca-bundle\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.088955 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.088933 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:34.157932 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.157875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-oauth-serving-cert\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.157932 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.157918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-serving-cert\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.158070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.157956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-oauth-config\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.158070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.157996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-service-ca\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.158070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.158020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45fsx\" (UniqueName: \"kubernetes.io/projected/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-kube-api-access-45fsx\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.158070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.158051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-trusted-ca-bundle\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.158258 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.158089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-config\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.158642 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.158619 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-service-ca\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.158866 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.158847 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-oauth-serving-cert\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.158931 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.158906 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-trusted-ca-bundle\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.159022 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.158994 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-config\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.159759 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.159732 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:34.160585 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.160564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-oauth-config\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.160691 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.160674 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-serving-cert\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.167081 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.167064 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45fsx\" (UniqueName: \"kubernetes.io/projected/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-kube-api-access-45fsx\") pod \"console-556ffd758f-8tzln\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.219592 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.219532 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:34.270749 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.270724 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:34.422900 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.422875 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556ffd758f-8tzln"] Aug 05 16:25:34.426304 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:25:34.426275 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64a4a2e_54f4_430e_b4f6_4cf162325eaf.slice/crio-87195a27f598d38c41482e88b330449ae8a87e64ffa12f98ee479f038edebabc WatchSource:0}: Error finding container 87195a27f598d38c41482e88b330449ae8a87e64ffa12f98ee479f038edebabc: Status 404 returned error can't find the container with id 87195a27f598d38c41482e88b330449ae8a87e64ffa12f98ee479f038edebabc Aug 05 16:25:34.770097 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.770046 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:34.833385 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.833350 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:34.850400 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.850367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556ffd758f-8tzln" event={"ID":"d64a4a2e-54f4-430e-b4f6-4cf162325eaf","Type":"ContainerStarted","Data":"afe30a53a7ffed1df10ffdc22f6d149f7c5f855e6241d93f3b0d6b64557bd0a7"} Aug 05 16:25:34.850768 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.850420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556ffd758f-8tzln" event={"ID":"d64a4a2e-54f4-430e-b4f6-4cf162325eaf","Type":"ContainerStarted","Data":"87195a27f598d38c41482e88b330449ae8a87e64ffa12f98ee479f038edebabc"} Aug 05 16:25:34.868726 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:34.868645 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-556ffd758f-8tzln" podStartSLOduration=1.868631371 podStartE2EDuration="1.868631371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:25:34.867483279 +0000 UTC m=+158.226735122" watchObservedRunningTime="2026-08-05 16:25:34.868631371 +0000 UTC m=+158.227883214" Aug 05 16:25:38.775693 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:38.775647 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:44.271843 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:44.271789 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:44.272247 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:44.271870 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:44.276588 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:44.276561 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:44.899654 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:44.899620 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:25:44.943862 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:44.943828 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-657cc76667-j5b2f"] Aug 05 16:25:45.095921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:45.095888 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:45.197149 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:25:45.197066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:09.964865 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:09.964805 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-657cc76667-j5b2f" podUID="c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" containerName="console" containerID="cri-o://23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0" gracePeriod=15 Aug 05 16:26:10.204148 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.204126 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:26:10.351679 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.351654 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-oauth-serving-cert\") pod \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " Aug 05 16:26:10.351823 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.351700 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-service-ca\") pod \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " Aug 05 16:26:10.351823 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.351735 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrqn2\" (UniqueName: \"kubernetes.io/projected/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-kube-api-access-rrqn2\") pod \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " Aug 05 16:26:10.351823 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.351759 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-trusted-ca-bundle\") pod \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " Aug 05 16:26:10.351823 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.351785 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-config\") pod \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " Aug 05 16:26:10.352006 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.351828 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-oauth-config\") pod \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " Aug 05 16:26:10.352006 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.351867 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-serving-cert\") pod \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\" (UID: \"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32\") " Aug 05 16:26:10.352147 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.352120 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" (UID: "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:26:10.352225 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.352200 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-service-ca" (OuterVolumeSpecName: "service-ca") pod "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" (UID: "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:26:10.352225 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.352209 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" (UID: "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:26:10.352318 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.352239 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-config" (OuterVolumeSpecName: "console-config") pod "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" (UID: "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:26:10.354193 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.354173 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" (UID: "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:26:10.354289 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.354204 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" (UID: "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:26:10.354289 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.354264 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-kube-api-access-rrqn2" (OuterVolumeSpecName: "kube-api-access-rrqn2") pod "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" (UID: "c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32"). InnerVolumeSpecName "kube-api-access-rrqn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:26:10.453333 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.453313 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-service-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:26:10.453433 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.453336 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrqn2\" (UniqueName: \"kubernetes.io/projected/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-kube-api-access-rrqn2\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:26:10.453433 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.453345 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-trusted-ca-bundle\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:26:10.453433 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.453354 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-config\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:26:10.453433 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.453362 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-oauth-config\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:26:10.453433 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.453369 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-console-serving-cert\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:26:10.453433 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.453377 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32-oauth-serving-cert\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:26:10.999423 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.999393 2575 generic.go:356] "Generic (PLEG): container finished" podID="c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" containerID="23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0" exitCode=0 Aug 05 16:26:10.999789 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.999436 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657cc76667-j5b2f" Aug 05 16:26:10.999789 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.999464 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657cc76667-j5b2f" event={"ID":"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32","Type":"ContainerDied","Data":"23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0"} Aug 05 16:26:10.999789 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.999501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657cc76667-j5b2f" event={"ID":"c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32","Type":"ContainerDied","Data":"e25e0c5a37a50b791f6835278a480ed6e7f553bc4d710ac9344116270a417ee2"} Aug 05 16:26:10.999789 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:10.999519 2575 scope.go:117] "RemoveContainer" containerID="23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0" Aug 05 16:26:11.007921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:11.007876 2575 scope.go:117] "RemoveContainer" containerID="23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0" Aug 05 16:26:11.008131 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:26:11.008111 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0\": container with ID starting with 23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0 not found: ID does not exist" containerID="23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0" Aug 05 16:26:11.008190 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:11.008143 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0"} err="failed to get container status \"23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0\": rpc error: code = NotFound desc = could not find container \"23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0\": container with ID starting with 23cb77261d2e7ed8c8bbc3a13b08d44a9ccc250a0a639b350bea8e94cff890d0 not found: ID does not exist" Aug 05 16:26:11.019805 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:11.019776 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-657cc76667-j5b2f"] Aug 05 16:26:11.023805 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:11.023770 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-657cc76667-j5b2f"] Aug 05 16:26:11.118568 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:11.118544 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" path="/var/lib/kubelet/pods/c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32/volumes" Aug 05 16:26:11.638412 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:11.638383 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:11.686956 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:11.686932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:15.754599 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:15.754570 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:15.846484 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:15.846460 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:40.566461 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:40.566427 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:40.619516 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:40.619489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:47.628722 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:47.628691 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:47.728970 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:47.728937 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:47.830383 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:47.830351 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:48.370571 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:48.370540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:48.422672 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:48.422648 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:48.594280 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:48.594237 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:52.492099 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:52.492067 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:52.592492 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:52.592463 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:58.025026 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:58.024989 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556ffd758f-8tzln"] Aug 05 16:26:58.125225 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:58.125192 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:58.227244 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:26:58.227217 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:03.221389 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:03.221347 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:03.367771 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:03.367741 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:23.045146 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.045045 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-556ffd758f-8tzln" podUID="d64a4a2e-54f4-430e-b4f6-4cf162325eaf" containerName="console" containerID="cri-o://afe30a53a7ffed1df10ffdc22f6d149f7c5f855e6241d93f3b0d6b64557bd0a7" gracePeriod=15 Aug 05 16:27:23.273843 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.273811 2575 generic.go:356] "Generic (PLEG): container finished" podID="d64a4a2e-54f4-430e-b4f6-4cf162325eaf" containerID="afe30a53a7ffed1df10ffdc22f6d149f7c5f855e6241d93f3b0d6b64557bd0a7" exitCode=0 Aug 05 16:27:23.273843 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.273828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556ffd758f-8tzln" event={"ID":"d64a4a2e-54f4-430e-b4f6-4cf162325eaf","Type":"ContainerDied","Data":"afe30a53a7ffed1df10ffdc22f6d149f7c5f855e6241d93f3b0d6b64557bd0a7"} Aug 05 16:27:23.288747 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.288725 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:27:23.416316 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416269 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-serving-cert\") pod \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " Aug 05 16:27:23.416316 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416298 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-oauth-config\") pod \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " Aug 05 16:27:23.416466 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416326 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-config\") pod \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " Aug 05 16:27:23.416466 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416363 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-service-ca\") pod \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " Aug 05 16:27:23.416466 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416395 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-trusted-ca-bundle\") pod \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " Aug 05 16:27:23.416466 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416422 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-oauth-serving-cert\") pod \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " Aug 05 16:27:23.416636 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416493 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45fsx\" (UniqueName: \"kubernetes.io/projected/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-kube-api-access-45fsx\") pod \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\" (UID: \"d64a4a2e-54f4-430e-b4f6-4cf162325eaf\") " Aug 05 16:27:23.416881 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416855 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-config" (OuterVolumeSpecName: "console-config") pod "d64a4a2e-54f4-430e-b4f6-4cf162325eaf" (UID: "d64a4a2e-54f4-430e-b4f6-4cf162325eaf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:27:23.417001 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416869 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-service-ca" (OuterVolumeSpecName: "service-ca") pod "d64a4a2e-54f4-430e-b4f6-4cf162325eaf" (UID: "d64a4a2e-54f4-430e-b4f6-4cf162325eaf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:27:23.417001 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416910 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d64a4a2e-54f4-430e-b4f6-4cf162325eaf" (UID: "d64a4a2e-54f4-430e-b4f6-4cf162325eaf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:27:23.417001 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.416930 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d64a4a2e-54f4-430e-b4f6-4cf162325eaf" (UID: "d64a4a2e-54f4-430e-b4f6-4cf162325eaf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:27:23.418568 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.418547 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d64a4a2e-54f4-430e-b4f6-4cf162325eaf" (UID: "d64a4a2e-54f4-430e-b4f6-4cf162325eaf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:27:23.418648 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.418565 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d64a4a2e-54f4-430e-b4f6-4cf162325eaf" (UID: "d64a4a2e-54f4-430e-b4f6-4cf162325eaf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:27:23.418843 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.418789 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-kube-api-access-45fsx" (OuterVolumeSpecName: "kube-api-access-45fsx") pod "d64a4a2e-54f4-430e-b4f6-4cf162325eaf" (UID: "d64a4a2e-54f4-430e-b4f6-4cf162325eaf"). InnerVolumeSpecName "kube-api-access-45fsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:27:23.517031 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.517001 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-serving-cert\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:27:23.517031 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.517030 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-oauth-config\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:27:23.517031 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.517040 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-console-config\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:27:23.517187 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.517049 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-service-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:27:23.517187 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.517056 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-trusted-ca-bundle\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:27:23.517187 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.517065 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-oauth-serving-cert\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:27:23.517187 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:23.517074 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45fsx\" (UniqueName: \"kubernetes.io/projected/d64a4a2e-54f4-430e-b4f6-4cf162325eaf-kube-api-access-45fsx\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:27:24.278970 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:24.278942 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556ffd758f-8tzln" Aug 05 16:27:24.278970 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:24.278951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556ffd758f-8tzln" event={"ID":"d64a4a2e-54f4-430e-b4f6-4cf162325eaf","Type":"ContainerDied","Data":"87195a27f598d38c41482e88b330449ae8a87e64ffa12f98ee479f038edebabc"} Aug 05 16:27:24.279436 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:24.279003 2575 scope.go:117] "RemoveContainer" containerID="afe30a53a7ffed1df10ffdc22f6d149f7c5f855e6241d93f3b0d6b64557bd0a7" Aug 05 16:27:24.301188 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:24.301168 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556ffd758f-8tzln"] Aug 05 16:27:24.305713 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:24.305691 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-556ffd758f-8tzln"] Aug 05 16:27:25.118945 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:25.118910 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d64a4a2e-54f4-430e-b4f6-4cf162325eaf" path="/var/lib/kubelet/pods/d64a4a2e-54f4-430e-b4f6-4cf162325eaf/volumes" Aug 05 16:27:33.799912 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:33.799880 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:57.007633 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:57.007598 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:27:57.011436 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:57.011403 2575 kubelet.go:1648] "Image garbage collection succeeded" Aug 05 16:27:57.012366 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:27:57.012350 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:28:28.567750 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:28.567717 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:30.472567 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:30.472534 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:50.192374 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:50.192293 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:50.984704 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:50.984668 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:53.810248 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:53.810216 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:53.910039 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:53.909989 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:56.567099 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:56.567068 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:56.655762 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:56.655732 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:56.892830 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:56.892713 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:56.999377 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:56.999343 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:58.707106 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:58.707059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:59.109577 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.109546 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:59.914772 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.914738 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k"] Aug 05 16:28:59.915221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.915071 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" containerName="console" Aug 05 16:28:59.915221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.915084 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" containerName="console" Aug 05 16:28:59.915221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.915093 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64a4a2e-54f4-430e-b4f6-4cf162325eaf" containerName="console" Aug 05 16:28:59.915221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.915099 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64a4a2e-54f4-430e-b4f6-4cf162325eaf" containerName="console" Aug 05 16:28:59.915221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.915148 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d64a4a2e-54f4-430e-b4f6-4cf162325eaf" containerName="console" Aug 05 16:28:59.915221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.915157 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9cd3ff5-2ba8-4f53-aded-dd5aa768aa32" containerName="console" Aug 05 16:28:59.918034 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.918017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" Aug 05 16:28:59.921157 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.921138 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 05 16:28:59.921406 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.921390 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 05 16:28:59.922549 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.922530 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-l9chm\"" Aug 05 16:28:59.924007 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:28:59.923990 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k"] Aug 05 16:29:00.014421 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.014383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzqn\" (UniqueName: \"kubernetes.io/projected/0378b34a-b24d-44dc-afb2-e7fdd6b4f247-kube-api-access-ltzqn\") pod \"cert-manager-cainjector-559cdd8c87-9kq2k\" (UID: \"0378b34a-b24d-44dc-afb2-e7fdd6b4f247\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" Aug 05 16:29:00.014581 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.014433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0378b34a-b24d-44dc-afb2-e7fdd6b4f247-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9kq2k\" (UID: \"0378b34a-b24d-44dc-afb2-e7fdd6b4f247\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" Aug 05 16:29:00.115863 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.115823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0378b34a-b24d-44dc-afb2-e7fdd6b4f247-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9kq2k\" (UID: \"0378b34a-b24d-44dc-afb2-e7fdd6b4f247\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" Aug 05 16:29:00.116036 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.115961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzqn\" (UniqueName: \"kubernetes.io/projected/0378b34a-b24d-44dc-afb2-e7fdd6b4f247-kube-api-access-ltzqn\") pod \"cert-manager-cainjector-559cdd8c87-9kq2k\" (UID: \"0378b34a-b24d-44dc-afb2-e7fdd6b4f247\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" Aug 05 16:29:00.124765 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.124727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0378b34a-b24d-44dc-afb2-e7fdd6b4f247-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9kq2k\" (UID: \"0378b34a-b24d-44dc-afb2-e7fdd6b4f247\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" Aug 05 16:29:00.124934 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.124803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzqn\" (UniqueName: \"kubernetes.io/projected/0378b34a-b24d-44dc-afb2-e7fdd6b4f247-kube-api-access-ltzqn\") pod \"cert-manager-cainjector-559cdd8c87-9kq2k\" (UID: \"0378b34a-b24d-44dc-afb2-e7fdd6b4f247\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" Aug 05 16:29:00.204530 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.204451 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:00.227559 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.227535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" Aug 05 16:29:00.263429 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.263386 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:00.356137 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.356108 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k"] Aug 05 16:29:00.358260 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:29:00.358234 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0378b34a_b24d_44dc_afb2_e7fdd6b4f247.slice/crio-f4271d789381c544cafbcdf828941875fed951d0a9b9435e36b00ab869738b31 WatchSource:0}: Error finding container f4271d789381c544cafbcdf828941875fed951d0a9b9435e36b00ab869738b31: Status 404 returned error can't find the container with id f4271d789381c544cafbcdf828941875fed951d0a9b9435e36b00ab869738b31 Aug 05 16:29:00.360233 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.360217 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:29:00.645363 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:00.645328 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" event={"ID":"0378b34a-b24d-44dc-afb2-e7fdd6b4f247","Type":"ContainerStarted","Data":"f4271d789381c544cafbcdf828941875fed951d0a9b9435e36b00ab869738b31"} Aug 05 16:29:03.658010 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:03.657971 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" event={"ID":"0378b34a-b24d-44dc-afb2-e7fdd6b4f247","Type":"ContainerStarted","Data":"7962858b35b7914070de4f9d67092b20997090d87b9b196e7c032aa976c8bfb6"} Aug 05 16:29:03.673966 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:03.673923 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9kq2k" podStartSLOduration=1.845378623 podStartE2EDuration="4.673909046s" totalImagesPullingTime="2.828530423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:29:03.673311369 +0000 UTC m=+367.032563212" watchObservedRunningTime="2026-08-05 16:29:03.673909046 +0000 UTC m=+367.033160889" Aug 05 16:29:03.960357 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:03.960281 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:04.015356 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:04.015330 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:07.519431 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:07.519391 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:07.892155 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:07.892087 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:07.994893 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:07.994868 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:09.622421 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.622392 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rqhsx"] Aug 05 16:29:09.625637 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.625622 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rqhsx" Aug 05 16:29:09.628252 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.628236 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kp8tk\"" Aug 05 16:29:09.640602 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.640585 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rqhsx"] Aug 05 16:29:09.792414 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.792382 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcq4l\" (UniqueName: \"kubernetes.io/projected/e4d29c46-06fb-4a7b-986d-5d07860d9e42-kube-api-access-mcq4l\") pod \"cert-manager-55b4d954d5-rqhsx\" (UID: \"e4d29c46-06fb-4a7b-986d-5d07860d9e42\") " pod="cert-manager/cert-manager-55b4d954d5-rqhsx" Aug 05 16:29:09.792588 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.792433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4d29c46-06fb-4a7b-986d-5d07860d9e42-bound-sa-token\") pod \"cert-manager-55b4d954d5-rqhsx\" (UID: \"e4d29c46-06fb-4a7b-986d-5d07860d9e42\") " pod="cert-manager/cert-manager-55b4d954d5-rqhsx" Aug 05 16:29:09.893730 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.893654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4d29c46-06fb-4a7b-986d-5d07860d9e42-bound-sa-token\") pod \"cert-manager-55b4d954d5-rqhsx\" (UID: \"e4d29c46-06fb-4a7b-986d-5d07860d9e42\") " pod="cert-manager/cert-manager-55b4d954d5-rqhsx" Aug 05 16:29:09.893873 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.893731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcq4l\" (UniqueName: \"kubernetes.io/projected/e4d29c46-06fb-4a7b-986d-5d07860d9e42-kube-api-access-mcq4l\") pod \"cert-manager-55b4d954d5-rqhsx\" (UID: \"e4d29c46-06fb-4a7b-986d-5d07860d9e42\") " pod="cert-manager/cert-manager-55b4d954d5-rqhsx" Aug 05 16:29:09.903862 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.903825 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4d29c46-06fb-4a7b-986d-5d07860d9e42-bound-sa-token\") pod \"cert-manager-55b4d954d5-rqhsx\" (UID: \"e4d29c46-06fb-4a7b-986d-5d07860d9e42\") " pod="cert-manager/cert-manager-55b4d954d5-rqhsx" Aug 05 16:29:09.904637 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.904617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcq4l\" (UniqueName: \"kubernetes.io/projected/e4d29c46-06fb-4a7b-986d-5d07860d9e42-kube-api-access-mcq4l\") pod \"cert-manager-55b4d954d5-rqhsx\" (UID: \"e4d29c46-06fb-4a7b-986d-5d07860d9e42\") " pod="cert-manager/cert-manager-55b4d954d5-rqhsx" Aug 05 16:29:09.934161 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:09.934140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rqhsx" Aug 05 16:29:10.056910 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:10.056867 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rqhsx"] Aug 05 16:29:10.059491 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:29:10.059456 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4d29c46_06fb_4a7b_986d_5d07860d9e42.slice/crio-55c5375d31c072d9f0c6f574b61182205337dc0a60630693f75ec38b895d0881 WatchSource:0}: Error finding container 55c5375d31c072d9f0c6f574b61182205337dc0a60630693f75ec38b895d0881: Status 404 returned error can't find the container with id 55c5375d31c072d9f0c6f574b61182205337dc0a60630693f75ec38b895d0881 Aug 05 16:29:10.685084 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:10.685049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rqhsx" event={"ID":"e4d29c46-06fb-4a7b-986d-5d07860d9e42","Type":"ContainerStarted","Data":"c43eba9fecd81d5e66ab7b657a1fd929eb28e8c83a6fe669f4ddffec34dfe299"} Aug 05 16:29:10.685084 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:10.685089 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rqhsx" event={"ID":"e4d29c46-06fb-4a7b-986d-5d07860d9e42","Type":"ContainerStarted","Data":"55c5375d31c072d9f0c6f574b61182205337dc0a60630693f75ec38b895d0881"} Aug 05 16:29:10.705441 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:10.705397 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rqhsx" podStartSLOduration=1.7053815559999999 podStartE2EDuration="1.705381556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:29:10.703348591 +0000 UTC m=+374.062600434" watchObservedRunningTime="2026-08-05 16:29:10.705381556 +0000 UTC m=+374.064633399" Aug 05 16:29:13.143239 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:13.143211 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:13.246877 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:13.246853 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:15.279348 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:15.279318 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:24.797855 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.797820 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bdc47958d-vvkq5"] Aug 05 16:29:24.802045 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.802026 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.804490 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.804459 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-trusted-ca-bundle\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.804490 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.804492 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/913c732e-f371-48b6-8b00-caa7f8264b88-console-oauth-config\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.804672 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.804515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/913c732e-f371-48b6-8b00-caa7f8264b88-console-serving-cert\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.804672 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.804585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-console-config\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.804672 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.804619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-service-ca\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.804672 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.804661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvgh\" (UniqueName: \"kubernetes.io/projected/913c732e-f371-48b6-8b00-caa7f8264b88-kube-api-access-bdvgh\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.804860 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.804707 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 05 16:29:24.804860 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.804712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-oauth-serving-cert\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.806007 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.805986 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 05 16:29:24.806121 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.805987 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6kwgg\"" Aug 05 16:29:24.806121 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.805992 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 05 16:29:24.806221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.806169 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 05 16:29:24.806334 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.806321 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 05 16:29:24.810506 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.810488 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 05 16:29:24.813174 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.813156 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdc47958d-vvkq5"] Aug 05 16:29:24.849064 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.849033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:24.905697 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.905668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-service-ca\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.905697 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.905701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvgh\" (UniqueName: \"kubernetes.io/projected/913c732e-f371-48b6-8b00-caa7f8264b88-kube-api-access-bdvgh\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.905952 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.905723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-oauth-serving-cert\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.905952 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.905746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-trusted-ca-bundle\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.905952 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.905787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/913c732e-f371-48b6-8b00-caa7f8264b88-console-oauth-config\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.905952 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.905860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/913c732e-f371-48b6-8b00-caa7f8264b88-console-serving-cert\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.905952 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.905921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-console-config\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.906527 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.906500 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-service-ca\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.906662 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.906639 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-oauth-serving-cert\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.906725 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.906658 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-trusted-ca-bundle\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.906725 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.906643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/913c732e-f371-48b6-8b00-caa7f8264b88-console-config\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.908378 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.908354 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/913c732e-f371-48b6-8b00-caa7f8264b88-console-oauth-config\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.908451 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.908402 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/913c732e-f371-48b6-8b00-caa7f8264b88-console-serving-cert\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:24.913817 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:24.913783 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvgh\" (UniqueName: \"kubernetes.io/projected/913c732e-f371-48b6-8b00-caa7f8264b88-kube-api-access-bdvgh\") pod \"console-6bdc47958d-vvkq5\" (UID: \"913c732e-f371-48b6-8b00-caa7f8264b88\") " pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:25.112868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:25.112780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:25.235313 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:25.235290 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdc47958d-vvkq5"] Aug 05 16:29:25.237177 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:29:25.237150 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913c732e_f371_48b6_8b00_caa7f8264b88.slice/crio-6e4c6598ea8a19715c244d252973351f19d10809febe375e02c74b63cd8fa284 WatchSource:0}: Error finding container 6e4c6598ea8a19715c244d252973351f19d10809febe375e02c74b63cd8fa284: Status 404 returned error can't find the container with id 6e4c6598ea8a19715c244d252973351f19d10809febe375e02c74b63cd8fa284 Aug 05 16:29:25.745473 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:25.745440 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdc47958d-vvkq5" event={"ID":"913c732e-f371-48b6-8b00-caa7f8264b88","Type":"ContainerStarted","Data":"7ebf8918e753d292d64699b93247a924ddafd45bce2ca5bbeb707f74b8b41c64"} Aug 05 16:29:25.745657 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:25.745490 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdc47958d-vvkq5" event={"ID":"913c732e-f371-48b6-8b00-caa7f8264b88","Type":"ContainerStarted","Data":"6e4c6598ea8a19715c244d252973351f19d10809febe375e02c74b63cd8fa284"} Aug 05 16:29:25.762620 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:25.762579 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bdc47958d-vvkq5" podStartSLOduration=1.762564289 podStartE2EDuration="1.762564289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:29:25.761940251 +0000 UTC m=+389.121192108" watchObservedRunningTime="2026-08-05 16:29:25.762564289 +0000 UTC m=+389.121816133" Aug 05 16:29:29.746608 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:29.746575 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:29.850470 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:29.850444 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:30.637875 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:30.637839 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:30.734030 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:30.733996 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:31.257329 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.257304 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:31.463098 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.463069 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:31.566607 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.566540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:31.619763 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.619736 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj"] Aug 05 16:29:31.623577 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.623558 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.626846 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.626829 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 05 16:29:31.627534 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.627515 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 05 16:29:31.627681 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.627668 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sjz7k\"" Aug 05 16:29:31.627736 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.627696 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 05 16:29:31.627781 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.627732 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 05 16:29:31.648419 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.648396 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj"] Aug 05 16:29:31.653970 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.653939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.654065 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.653994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-manifests\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.654065 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.654024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.654065 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.654052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qpv\" (UniqueName: \"kubernetes.io/projected/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-kube-api-access-j6qpv\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.654166 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.654137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-tmp\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.755147 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.755119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-tmp\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.755237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.755175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.755237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.755196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-manifests\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.755237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.755223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.755360 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.755242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qpv\" (UniqueName: \"kubernetes.io/projected/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-kube-api-access-j6qpv\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.755578 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.755549 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-manifests\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.757529 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.757509 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-tmp\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.757727 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.757711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.757774 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.757724 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.767506 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.767483 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qpv\" (UniqueName: \"kubernetes.io/projected/8b5c41a9-f9d5-4844-8580-26c0f3f53edc-kube-api-access-j6qpv\") pod \"opendatahub-operator-controller-manager-5cfc779b6-f6hsj\" (UID: \"8b5c41a9-f9d5-4844-8580-26c0f3f53edc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:31.801539 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.801515 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:31.902240 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.902182 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:31.934498 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:31.934477 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:32.004898 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:32.004855 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:32.066723 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:32.066695 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj"] Aug 05 16:29:32.070291 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:29:32.070227 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5c41a9_f9d5_4844_8580_26c0f3f53edc.slice/crio-a0196d5c7d4e86745c501187ece52c975fae236521c60c5ed675f4814c433b62 WatchSource:0}: Error finding container a0196d5c7d4e86745c501187ece52c975fae236521c60c5ed675f4814c433b62: Status 404 returned error can't find the container with id a0196d5c7d4e86745c501187ece52c975fae236521c60c5ed675f4814c433b62 Aug 05 16:29:32.071235 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:32.071194 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:32.140348 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:32.140299 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:32.243110 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:32.243087 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:32.340639 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:32.340611 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:32.775979 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:32.775941 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" event={"ID":"8b5c41a9-f9d5-4844-8580-26c0f3f53edc","Type":"ContainerStarted","Data":"a0196d5c7d4e86745c501187ece52c975fae236521c60c5ed675f4814c433b62"} Aug 05 16:29:34.648321 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:34.648291 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:34.748734 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:34.748708 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:35.119424 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:35.119396 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:35.119596 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:35.119498 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:35.119596 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:35.119521 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:35.122993 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:35.122975 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bdc47958d-vvkq5" Aug 05 16:29:35.299913 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:35.299884 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:35.417460 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:35.417404 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:35.507762 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:35.507734 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:35.607759 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:35.607734 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:35.791680 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:35.791653 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" event={"ID":"8b5c41a9-f9d5-4844-8580-26c0f3f53edc","Type":"ContainerStarted","Data":"94e15c9026d8de6a66d860ae6513ba6d7b713e8013b7799c4ecc20e7a7be85f3"} Aug 05 16:29:36.797037 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:36.796991 2575 generic.go:356] "Generic (PLEG): container finished" podID="8b5c41a9-f9d5-4844-8580-26c0f3f53edc" containerID="94e15c9026d8de6a66d860ae6513ba6d7b713e8013b7799c4ecc20e7a7be85f3" exitCode=0 Aug 05 16:29:36.797407 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:36.797075 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" event={"ID":"8b5c41a9-f9d5-4844-8580-26c0f3f53edc","Type":"ContainerDied","Data":"94e15c9026d8de6a66d860ae6513ba6d7b713e8013b7799c4ecc20e7a7be85f3"} Aug 05 16:29:37.804710 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:37.804612 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" event={"ID":"8b5c41a9-f9d5-4844-8580-26c0f3f53edc","Type":"ContainerStarted","Data":"ffb2ddf11c562c0394adf3ea19f986e64f0d6fbd9edbb41b94feb5a3d1dd88cb"} Aug 05 16:29:37.804710 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:37.804692 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:37.828414 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:37.828373 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" podStartSLOduration=2.702352366 podStartE2EDuration="6.82835729s" totalImagesPullingTime="3.095501242s" totalInitContainerRuntime="1.030503682s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:29:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:29:37.826178642 +0000 UTC m=+401.185430488" watchObservedRunningTime="2026-08-05 16:29:37.82835729 +0000 UTC m=+401.187609209" Aug 05 16:29:41.639377 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:41.639343 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:41.743713 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:41.743682 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:44.791162 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:44.791126 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:47.551386 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:47.551354 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:47.614181 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:47.614149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:47.979518 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:47.979441 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:48.812127 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:48.812098 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-f6hsj" Aug 05 16:29:50.432046 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:50.432014 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:50.563397 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:50.563364 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:51.033565 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.033530 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:51.035667 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.035637 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65879f568d-tpmh6"] Aug 05 16:29:51.037968 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.037953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.040631 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.040614 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 05 16:29:51.040734 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.040614 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 05 16:29:51.040734 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.040614 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 05 16:29:51.041056 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.041035 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 05 16:29:51.057819 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.057777 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65879f568d-tpmh6"] Aug 05 16:29:51.108610 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.108586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/098f0277-617a-4761-bfed-e583eab271ff-tmp\") pod \"kube-auth-proxy-65879f568d-tpmh6\" (UID: \"098f0277-617a-4761-bfed-e583eab271ff\") " pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.108699 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.108615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098f0277-617a-4761-bfed-e583eab271ff-tls-certs\") pod \"kube-auth-proxy-65879f568d-tpmh6\" (UID: \"098f0277-617a-4761-bfed-e583eab271ff\") " pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.108699 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.108686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2k6\" (UniqueName: \"kubernetes.io/projected/098f0277-617a-4761-bfed-e583eab271ff-kube-api-access-mq2k6\") pod \"kube-auth-proxy-65879f568d-tpmh6\" (UID: \"098f0277-617a-4761-bfed-e583eab271ff\") " pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.209710 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.209682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2k6\" (UniqueName: \"kubernetes.io/projected/098f0277-617a-4761-bfed-e583eab271ff-kube-api-access-mq2k6\") pod \"kube-auth-proxy-65879f568d-tpmh6\" (UID: \"098f0277-617a-4761-bfed-e583eab271ff\") " pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.209858 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.209767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/098f0277-617a-4761-bfed-e583eab271ff-tmp\") pod \"kube-auth-proxy-65879f568d-tpmh6\" (UID: \"098f0277-617a-4761-bfed-e583eab271ff\") " pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.209858 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.209814 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098f0277-617a-4761-bfed-e583eab271ff-tls-certs\") pod \"kube-auth-proxy-65879f568d-tpmh6\" (UID: \"098f0277-617a-4761-bfed-e583eab271ff\") " pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.209978 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:29:51.209939 2575 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Aug 05 16:29:51.210029 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:29:51.210010 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098f0277-617a-4761-bfed-e583eab271ff-tls-certs podName:098f0277-617a-4761-bfed-e583eab271ff nodeName:}" failed. No retries permitted until 2026-08-05 16:29:51.709988543 +0000 UTC m=+415.069240367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/098f0277-617a-4761-bfed-e583eab271ff-tls-certs") pod "kube-auth-proxy-65879f568d-tpmh6" (UID: "098f0277-617a-4761-bfed-e583eab271ff") : secret "kube-auth-proxy-tls" not found Aug 05 16:29:51.212321 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.212299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/098f0277-617a-4761-bfed-e583eab271ff-tmp\") pod \"kube-auth-proxy-65879f568d-tpmh6\" (UID: \"098f0277-617a-4761-bfed-e583eab271ff\") " pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.225993 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.225965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2k6\" (UniqueName: \"kubernetes.io/projected/098f0277-617a-4761-bfed-e583eab271ff-kube-api-access-mq2k6\") pod \"kube-auth-proxy-65879f568d-tpmh6\" (UID: \"098f0277-617a-4761-bfed-e583eab271ff\") " pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.230214 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.230179 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:51.328916 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.328861 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:51.429096 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.429073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:51.513602 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.513579 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:51.714961 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.714891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098f0277-617a-4761-bfed-e583eab271ff-tls-certs\") pod \"kube-auth-proxy-65879f568d-tpmh6\" (UID: \"098f0277-617a-4761-bfed-e583eab271ff\") " pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.717344 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.717318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098f0277-617a-4761-bfed-e583eab271ff-tls-certs\") pod \"kube-auth-proxy-65879f568d-tpmh6\" (UID: \"098f0277-617a-4761-bfed-e583eab271ff\") " pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.948330 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.948305 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" Aug 05 16:29:51.969408 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:51.969341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:52.066028 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:52.066001 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:52.083969 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:52.083948 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65879f568d-tpmh6"] Aug 05 16:29:52.086365 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:29:52.086335 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098f0277_617a_4761_bfed_e583eab271ff.slice/crio-8c267c301b9342f7b8eba923cd4abd4ef02f01cf990badce4c26e7e9292a260e WatchSource:0}: Error finding container 8c267c301b9342f7b8eba923cd4abd4ef02f01cf990badce4c26e7e9292a260e: Status 404 returned error can't find the container with id 8c267c301b9342f7b8eba923cd4abd4ef02f01cf990badce4c26e7e9292a260e Aug 05 16:29:52.470144 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:52.470111 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:52.575331 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:52.575300 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:52.879828 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:52.879764 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" event={"ID":"098f0277-617a-4761-bfed-e583eab271ff","Type":"ContainerStarted","Data":"8c267c301b9342f7b8eba923cd4abd4ef02f01cf990badce4c26e7e9292a260e"} Aug 05 16:29:53.481562 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:53.481527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:54.891358 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:54.891331 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" event={"ID":"098f0277-617a-4761-bfed-e583eab271ff","Type":"ContainerStarted","Data":"fac60a66cdd044d499386401cfe8b30d6df46d7d17e2899072ac3995ffba8850"} Aug 05 16:29:54.908715 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:54.908663 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65879f568d-tpmh6" podStartSLOduration=1.16382075 podStartE2EDuration="3.908644182s" totalImagesPullingTime="2.744823432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:29:54.90751748 +0000 UTC m=+418.266769318" watchObservedRunningTime="2026-08-05 16:29:54.908644182 +0000 UTC m=+418.267896027" Aug 05 16:29:57.346164 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.346131 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:57.445108 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.445074 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:57.579372 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.579342 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd"] Aug 05 16:29:57.582850 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.582830 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.591380 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.586081 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 05 16:29:57.591380 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.586182 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-c7zt7\"" Aug 05 16:29:57.596922 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.596856 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd"] Aug 05 16:29:57.616120 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.616091 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:57.640586 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.640562 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f"] Aug 05 16:29:57.643069 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.643052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.655044 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.655024 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f"] Aug 05 16:29:57.661091 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.661071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.661185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.661099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.661185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.661118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.661185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.661145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.661321 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.661251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8t6p\" (UniqueName: \"kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-kube-api-access-r8t6p\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.661321 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.661279 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.661393 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.661321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.661393 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.661339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.661393 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.661362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.762681 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.762681 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.762900 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.762900 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.762900 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762744 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.762900 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762766 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.762900 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762858 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8t6p\" (UniqueName: \"kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-kube-api-access-r8t6p\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.762900 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762967 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.762990 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tbtc\" (UniqueName: \"kubernetes.io/projected/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-kube-api-access-7tbtc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763117 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763158 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.763185 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763187 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.763661 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.763661 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763391 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.763661 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763505 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.763760 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.763742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.765280 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.765259 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.765600 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.765584 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.770854 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.770829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.771202 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.771180 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8t6p\" (UniqueName: \"kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-kube-api-access-r8t6p\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bckhtwd\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.829467 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.829439 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:57.863676 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.863621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tbtc\" (UniqueName: \"kubernetes.io/projected/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-kube-api-access-7tbtc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.863676 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.863651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.863676 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.863672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.863856 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.863787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.863856 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.863849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.863937 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.863887 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.863937 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.863921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.864034 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.863956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.864034 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.864000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.864177 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.864150 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.864253 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.864180 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.864335 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.864309 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.864335 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.864323 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.864661 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.864537 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.866023 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.866003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.866288 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.866272 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.872495 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.872474 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tbtc\" (UniqueName: \"kubernetes.io/projected/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-kube-api-access-7tbtc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.872590 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.872569 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15812cb0-a4b0-45c3-aeb1-7c73223b99c9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f\" (UID: \"15812cb0-a4b0-45c3-aeb1-7c73223b99c9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:57.900856 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.900824 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:29:57.935062 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.935033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:57.952743 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:57.952718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:29:58.072235 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:58.072209 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd"] Aug 05 16:29:58.074367 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:29:58.074344 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e9d54e_6bca_43dc_bba6_57ab8d668c10.slice/crio-0eb18c3493ff922d1eeffd0c1a44947eeda05aa9c5fd265cccf856af15a30b26 WatchSource:0}: Error finding container 0eb18c3493ff922d1eeffd0c1a44947eeda05aa9c5fd265cccf856af15a30b26: Status 404 returned error can't find the container with id 0eb18c3493ff922d1eeffd0c1a44947eeda05aa9c5fd265cccf856af15a30b26 Aug 05 16:29:58.099574 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:58.099546 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f"] Aug 05 16:29:58.101878 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:29:58.101855 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15812cb0_a4b0_45c3_aeb1_7c73223b99c9.slice/crio-d89250d5f3e342c2627eafc6046240ecbed191eedcb07ec3bba9c5b5e42a12d5 WatchSource:0}: Error finding container d89250d5f3e342c2627eafc6046240ecbed191eedcb07ec3bba9c5b5e42a12d5: Status 404 returned error can't find the container with id d89250d5f3e342c2627eafc6046240ecbed191eedcb07ec3bba9c5b5e42a12d5 Aug 05 16:29:58.909886 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:58.909858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" event={"ID":"15812cb0-a4b0-45c3-aeb1-7c73223b99c9","Type":"ContainerStarted","Data":"d89250d5f3e342c2627eafc6046240ecbed191eedcb07ec3bba9c5b5e42a12d5"} Aug 05 16:29:58.911043 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:58.911017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" event={"ID":"00e9d54e-6bca-43dc-bba6-57ab8d668c10","Type":"ContainerStarted","Data":"0eb18c3493ff922d1eeffd0c1a44947eeda05aa9c5fd265cccf856af15a30b26"} Aug 05 16:29:59.596221 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:29:59.596149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:00.066054 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.066021 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r"] Aug 05 16:30:00.069542 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.069517 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.072620 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.072597 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 05 16:30:00.072740 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.072631 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:30:00.073906 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.073885 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 05 16:30:00.074583 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.074088 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 05 16:30:00.074583 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.074113 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-49q4h\"" Aug 05 16:30:00.074583 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.074091 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 05 16:30:00.082150 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.082129 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r"] Aug 05 16:30:00.185682 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.185649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/66510f0e-ec14-4184-a04d-a13767463593-manager-config\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.185882 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.185691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/66510f0e-ec14-4184-a04d-a13767463593-metrics-cert\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.185945 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.185891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66510f0e-ec14-4184-a04d-a13767463593-cert\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.185994 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.185952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djkt\" (UniqueName: \"kubernetes.io/projected/66510f0e-ec14-4184-a04d-a13767463593-kube-api-access-7djkt\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.286923 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.286879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/66510f0e-ec14-4184-a04d-a13767463593-manager-config\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.286923 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.286922 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/66510f0e-ec14-4184-a04d-a13767463593-metrics-cert\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.287155 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.287032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66510f0e-ec14-4184-a04d-a13767463593-cert\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.287155 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.287071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djkt\" (UniqueName: \"kubernetes.io/projected/66510f0e-ec14-4184-a04d-a13767463593-kube-api-access-7djkt\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.287604 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.287571 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/66510f0e-ec14-4184-a04d-a13767463593-manager-config\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.290269 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.290243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/66510f0e-ec14-4184-a04d-a13767463593-metrics-cert\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.290905 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.290863 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66510f0e-ec14-4184-a04d-a13767463593-cert\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.305237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.305213 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djkt\" (UniqueName: \"kubernetes.io/projected/66510f0e-ec14-4184-a04d-a13767463593-kube-api-access-7djkt\") pod \"lws-controller-manager-6bfcf8b475-cs66r\" (UID: \"66510f0e-ec14-4184-a04d-a13767463593\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.382062 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.381995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:00.541182 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.541105 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r"] Aug 05 16:30:00.545600 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:30:00.545571 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66510f0e_ec14_4184_a04d_a13767463593.slice/crio-d6c99f91931c3fbc83582845899e3935a14de318c1a3295e2d727210e470ab6d WatchSource:0}: Error finding container d6c99f91931c3fbc83582845899e3935a14de318c1a3295e2d727210e470ab6d: Status 404 returned error can't find the container with id d6c99f91931c3fbc83582845899e3935a14de318c1a3295e2d727210e470ab6d Aug 05 16:30:00.923130 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:00.923089 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" event={"ID":"66510f0e-ec14-4184-a04d-a13767463593","Type":"ContainerStarted","Data":"d6c99f91931c3fbc83582845899e3935a14de318c1a3295e2d727210e470ab6d"} Aug 05 16:30:01.697912 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:01.697873 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 16:30:01.698165 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:01.697953 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 16:30:01.698165 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:01.697982 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 16:30:01.704213 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:01.704186 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 16:30:01.704280 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:01.704250 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 16:30:01.704311 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:01.704277 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 16:30:01.890922 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:01.890836 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:01.959302 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:01.958345 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" event={"ID":"15812cb0-a4b0-45c3-aeb1-7c73223b99c9","Type":"ContainerStarted","Data":"872a16fca7c008531d47526f4cd8c340fd3f7b51f00de0a34f92a1ae9a1dbbaf"} Aug 05 16:30:01.961906 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:01.961315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" event={"ID":"00e9d54e-6bca-43dc-bba6-57ab8d668c10","Type":"ContainerStarted","Data":"a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9"} Aug 05 16:30:02.026001 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.025940 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" podStartSLOduration=1.398470911 podStartE2EDuration="5.025917904s" totalImagesPullingTime="3.627446993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:30:02.024007336 +0000 UTC m=+425.383259178" watchObservedRunningTime="2026-08-05 16:30:02.025917904 +0000 UTC m=+425.385169750" Aug 05 16:30:02.026465 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.026433 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" podStartSLOduration=1.432516105 podStartE2EDuration="5.026424495s" totalImagesPullingTime="3.59390839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:30:01.996902932 +0000 UTC m=+425.356154768" watchObservedRunningTime="2026-08-05 16:30:02.026424495 +0000 UTC m=+425.385676341" Aug 05 16:30:02.078571 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.078294 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:02.214686 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.214570 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:02.901896 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.901857 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:30:02.903399 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.903373 2575 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Aug 05 16:30:02.903511 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.903422 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" podUID="00e9d54e-6bca-43dc-bba6-57ab8d668c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Aug 05 16:30:02.953416 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.953381 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:30:02.959200 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.959177 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:30:02.967024 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.966999 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:30:02.967838 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:02.967819 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f" Aug 05 16:30:03.043430 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:03.043397 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd"] Aug 05 16:30:03.246037 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:03.245992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:03.382637 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:03.382603 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:03.483465 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:03.483431 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:03.902320 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:03.902286 2575 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Aug 05 16:30:03.902728 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:03.902367 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" podUID="00e9d54e-6bca-43dc-bba6-57ab8d668c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Aug 05 16:30:04.902155 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:04.902119 2575 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Aug 05 16:30:04.902338 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:04.902177 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" podUID="00e9d54e-6bca-43dc-bba6-57ab8d668c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Aug 05 16:30:04.978296 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:04.978261 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" event={"ID":"66510f0e-ec14-4184-a04d-a13767463593","Type":"ContainerStarted","Data":"275b0cf4c42d9453eca64a2bbeb8fca74caf145e8989bcde495548dabb8d0dd3"} Aug 05 16:30:04.978473 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:04.978319 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" podUID="00e9d54e-6bca-43dc-bba6-57ab8d668c10" containerName="istio-proxy" containerID="cri-o://a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9" gracePeriod=30 Aug 05 16:30:05.982637 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:05.982608 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:06.445600 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:06.445559 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:06.505771 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:06.505743 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:10.221646 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.221621 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:30:10.247905 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.247849 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" podStartSLOduration=6.324496376 podStartE2EDuration="10.247828826s" totalImagesPullingTime="3.92333245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:30:05.021106849 +0000 UTC m=+428.380358694" watchObservedRunningTime="2026-08-05 16:30:10.247828826 +0000 UTC m=+433.607080670" Aug 05 16:30:10.392201 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392173 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-envoy\") pod \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " Aug 05 16:30:10.392351 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392263 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-podinfo\") pod \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " Aug 05 16:30:10.392351 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392283 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8t6p\" (UniqueName: \"kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-kube-api-access-r8t6p\") pod \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " Aug 05 16:30:10.392351 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392312 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-socket\") pod \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " Aug 05 16:30:10.392351 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-certs\") pod \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " Aug 05 16:30:10.392351 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392349 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istiod-ca-cert\") pod \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " Aug 05 16:30:10.392566 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392380 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-credential-socket\") pod \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " Aug 05 16:30:10.392566 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392405 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-token\") pod \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " Aug 05 16:30:10.392566 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392426 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-data\") pod \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\" (UID: \"00e9d54e-6bca-43dc-bba6-57ab8d668c10\") " Aug 05 16:30:10.392715 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392628 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00e9d54e-6bca-43dc-bba6-57ab8d668c10" (UID: "00e9d54e-6bca-43dc-bba6-57ab8d668c10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:30:10.392715 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392642 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00e9d54e-6bca-43dc-bba6-57ab8d668c10" (UID: "00e9d54e-6bca-43dc-bba6-57ab8d668c10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:30:10.392858 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392723 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-socket\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:30:10.392858 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392741 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-workload-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:30:10.392965 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392868 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-data" (OuterVolumeSpecName: "istio-data") pod "00e9d54e-6bca-43dc-bba6-57ab8d668c10" (UID: "00e9d54e-6bca-43dc-bba6-57ab8d668c10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:30:10.392965 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.392882 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00e9d54e-6bca-43dc-bba6-57ab8d668c10" (UID: "00e9d54e-6bca-43dc-bba6-57ab8d668c10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:30:10.393056 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.393030 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00e9d54e-6bca-43dc-bba6-57ab8d668c10" (UID: "00e9d54e-6bca-43dc-bba6-57ab8d668c10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:30:10.394545 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.394522 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00e9d54e-6bca-43dc-bba6-57ab8d668c10" (UID: "00e9d54e-6bca-43dc-bba6-57ab8d668c10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:30:10.394701 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.394687 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-token" (OuterVolumeSpecName: "istio-token") pod "00e9d54e-6bca-43dc-bba6-57ab8d668c10" (UID: "00e9d54e-6bca-43dc-bba6-57ab8d668c10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:30:10.395019 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.394997 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00e9d54e-6bca-43dc-bba6-57ab8d668c10" (UID: "00e9d54e-6bca-43dc-bba6-57ab8d668c10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 16:30:10.395019 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.395000 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-kube-api-access-r8t6p" (OuterVolumeSpecName: "kube-api-access-r8t6p") pod "00e9d54e-6bca-43dc-bba6-57ab8d668c10" (UID: "00e9d54e-6bca-43dc-bba6-57ab8d668c10"). InnerVolumeSpecName "kube-api-access-r8t6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:30:10.493417 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.493390 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istiod-ca-cert\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:30:10.493417 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.493413 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-credential-socket\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:30:10.493417 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.493422 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-token\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:30:10.493607 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.493431 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-data\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:30:10.493607 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.493438 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-envoy\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:30:10.493607 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.493445 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e9d54e-6bca-43dc-bba6-57ab8d668c10-istio-podinfo\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:30:10.493607 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:10.493453 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8t6p\" (UniqueName: \"kubernetes.io/projected/00e9d54e-6bca-43dc-bba6-57ab8d668c10-kube-api-access-r8t6p\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:30:11.005152 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:11.005124 2575 generic.go:356] "Generic (PLEG): container finished" podID="00e9d54e-6bca-43dc-bba6-57ab8d668c10" containerID="a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9" exitCode=0 Aug 05 16:30:11.005295 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:11.005159 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" event={"ID":"00e9d54e-6bca-43dc-bba6-57ab8d668c10","Type":"ContainerDied","Data":"a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9"} Aug 05 16:30:11.005295 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:11.005167 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" Aug 05 16:30:11.005295 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:11.005197 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd" event={"ID":"00e9d54e-6bca-43dc-bba6-57ab8d668c10","Type":"ContainerDied","Data":"0eb18c3493ff922d1eeffd0c1a44947eeda05aa9c5fd265cccf856af15a30b26"} Aug 05 16:30:11.005295 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:11.005221 2575 scope.go:117] "RemoveContainer" containerID="a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9" Aug 05 16:30:11.014161 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:11.014144 2575 scope.go:117] "RemoveContainer" containerID="a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9" Aug 05 16:30:11.014394 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:30:11.014377 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9\": container with ID starting with a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9 not found: ID does not exist" containerID="a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9" Aug 05 16:30:11.014441 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:11.014403 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9"} err="failed to get container status \"a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9\": rpc error: code = NotFound desc = could not find container \"a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9\": container with ID starting with a52d9c6a21ce512da7df80248290b8f7ef95048808fb95f030783e661f3a89d9 not found: ID does not exist" Aug 05 16:30:11.025970 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:11.025946 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd"] Aug 05 16:30:11.030449 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:11.030430 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bckhtwd"] Aug 05 16:30:11.119313 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:11.119286 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e9d54e-6bca-43dc-bba6-57ab8d668c10" path="/var/lib/kubelet/pods/00e9d54e-6bca-43dc-bba6-57ab8d668c10/volumes" Aug 05 16:30:16.989651 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:16.989622 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-cs66r" Aug 05 16:30:52.062469 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:52.062429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:52.369569 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:30:52.369488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:04.343659 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:04.343630 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:05.131346 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:05.131320 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:08.205768 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:08.205720 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:08.457127 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:08.457049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:10.136907 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:10.136879 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:10.391975 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:10.391898 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:10.542365 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:10.542338 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:11.184651 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:11.184617 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:11.534474 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:11.534451 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:11.633152 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:11.633118 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:14.566706 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:14.566679 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:15.599916 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:15.599876 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:17.544415 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:17.544365 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:17.655442 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:17.655412 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:17.945756 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:17.945692 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:18.043099 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:18.043063 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:21.160402 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:21.160371 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:21.399339 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:21.399308 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:21.607011 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:21.606982 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:21.707114 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:21.707093 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:25.909681 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:25.909652 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:26.006741 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:26.006713 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:26.067033 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:26.067008 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:32.451987 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:32.451955 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:33.592327 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:33.592298 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:37.942580 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:37.942546 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:38.002289 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:38.002265 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:38.545079 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:38.545048 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:38.602598 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:38.602568 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:41.943993 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:41.943965 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:42.044400 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:42.044374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:46.170641 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:46.170608 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:46.273269 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:46.273236 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:46.389940 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:46.389912 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:46.490220 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:46.490161 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:46.593755 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:46.593727 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:46.693332 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:46.693304 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:52.456276 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:52.456194 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:52.515249 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:31:52.515224 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:00.890187 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:00.890152 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:01.221033 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:01.220952 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:17.732413 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.732377 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:32:17.732958 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.732919 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e9d54e-6bca-43dc-bba6-57ab8d668c10" containerName="istio-proxy" Aug 05 16:32:17.732958 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.732938 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e9d54e-6bca-43dc-bba6-57ab8d668c10" containerName="istio-proxy" Aug 05 16:32:17.733067 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.733047 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e9d54e-6bca-43dc-bba6-57ab8d668c10" containerName="istio-proxy" Aug 05 16:32:17.735155 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.735136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" Aug 05 16:32:17.738122 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.738100 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 05 16:32:17.738237 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.738104 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 05 16:32:17.739398 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.739375 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fxgch\"" Aug 05 16:32:17.739511 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.739383 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 05 16:32:17.745716 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.745695 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:32:17.777778 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.777744 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:32:17.867685 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.867645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67htf\" (UniqueName: \"kubernetes.io/projected/d8b8fa28-e50c-4a67-9f58-13df531c13c7-kube-api-access-67htf\") pod \"limitador-limitador-85d9db4d5d-zf2hr\" (UID: \"d8b8fa28-e50c-4a67-9f58-13df531c13c7\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" Aug 05 16:32:17.867883 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.867696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d8b8fa28-e50c-4a67-9f58-13df531c13c7-config-file\") pod \"limitador-limitador-85d9db4d5d-zf2hr\" (UID: \"d8b8fa28-e50c-4a67-9f58-13df531c13c7\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" Aug 05 16:32:17.968168 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.968137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67htf\" (UniqueName: \"kubernetes.io/projected/d8b8fa28-e50c-4a67-9f58-13df531c13c7-kube-api-access-67htf\") pod \"limitador-limitador-85d9db4d5d-zf2hr\" (UID: \"d8b8fa28-e50c-4a67-9f58-13df531c13c7\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" Aug 05 16:32:17.968310 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.968191 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d8b8fa28-e50c-4a67-9f58-13df531c13c7-config-file\") pod \"limitador-limitador-85d9db4d5d-zf2hr\" (UID: \"d8b8fa28-e50c-4a67-9f58-13df531c13c7\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" Aug 05 16:32:17.968880 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.968858 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d8b8fa28-e50c-4a67-9f58-13df531c13c7-config-file\") pod \"limitador-limitador-85d9db4d5d-zf2hr\" (UID: \"d8b8fa28-e50c-4a67-9f58-13df531c13c7\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" Aug 05 16:32:17.976917 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:17.976892 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67htf\" (UniqueName: \"kubernetes.io/projected/d8b8fa28-e50c-4a67-9f58-13df531c13c7-kube-api-access-67htf\") pod \"limitador-limitador-85d9db4d5d-zf2hr\" (UID: \"d8b8fa28-e50c-4a67-9f58-13df531c13c7\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" Aug 05 16:32:18.036280 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.036248 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:18.045663 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.045633 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" Aug 05 16:32:18.209913 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.209890 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:32:18.211839 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:32:18.211788 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b8fa28_e50c_4a67_9f58_13df531c13c7.slice/crio-b95078fce8d39c2b16092c030ecdb4ed4b8829ae28bbeaace5beb1889f914c07 WatchSource:0}: Error finding container b95078fce8d39c2b16092c030ecdb4ed4b8829ae28bbeaace5beb1889f914c07: Status 404 returned error can't find the container with id b95078fce8d39c2b16092c030ecdb4ed4b8829ae28bbeaace5beb1889f914c07 Aug 05 16:32:18.346371 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.346292 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vq474"] Aug 05 16:32:18.349619 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.349603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vq474" Aug 05 16:32:18.352844 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.352817 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mh5mq\"" Aug 05 16:32:18.359338 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.359316 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vq474"] Aug 05 16:32:18.472565 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.472533 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sqnl\" (UniqueName: \"kubernetes.io/projected/6d962e48-deee-46e7-b198-c8c29ab0f258-kube-api-access-5sqnl\") pod \"authorino-c548b5d7d-vq474\" (UID: \"6d962e48-deee-46e7-b198-c8c29ab0f258\") " pod="kuadrant-system/authorino-c548b5d7d-vq474" Aug 05 16:32:18.556320 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.556284 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" event={"ID":"d8b8fa28-e50c-4a67-9f58-13df531c13c7","Type":"ContainerStarted","Data":"b95078fce8d39c2b16092c030ecdb4ed4b8829ae28bbeaace5beb1889f914c07"} Aug 05 16:32:18.573734 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.573692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sqnl\" (UniqueName: \"kubernetes.io/projected/6d962e48-deee-46e7-b198-c8c29ab0f258-kube-api-access-5sqnl\") pod \"authorino-c548b5d7d-vq474\" (UID: \"6d962e48-deee-46e7-b198-c8c29ab0f258\") " pod="kuadrant-system/authorino-c548b5d7d-vq474" Aug 05 16:32:18.582634 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.582609 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sqnl\" (UniqueName: \"kubernetes.io/projected/6d962e48-deee-46e7-b198-c8c29ab0f258-kube-api-access-5sqnl\") pod \"authorino-c548b5d7d-vq474\" (UID: \"6d962e48-deee-46e7-b198-c8c29ab0f258\") " pod="kuadrant-system/authorino-c548b5d7d-vq474" Aug 05 16:32:18.659253 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.659172 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vq474" Aug 05 16:32:18.774066 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.774034 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:18.786569 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.786541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vq474"] Aug 05 16:32:18.788918 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:32:18.788893 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d962e48_deee_46e7_b198_c8c29ab0f258.slice/crio-f7641868cb74bb6c0312643e9491d1de1c5aeb8e980e9138deef4ca3d1c3cb3f WatchSource:0}: Error finding container f7641868cb74bb6c0312643e9491d1de1c5aeb8e980e9138deef4ca3d1c3cb3f: Status 404 returned error can't find the container with id f7641868cb74bb6c0312643e9491d1de1c5aeb8e980e9138deef4ca3d1c3cb3f Aug 05 16:32:18.875353 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.875316 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:18.994193 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:18.994158 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:19.095140 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:19.095091 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:19.565290 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:19.565253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vq474" event={"ID":"6d962e48-deee-46e7-b198-c8c29ab0f258","Type":"ContainerStarted","Data":"f7641868cb74bb6c0312643e9491d1de1c5aeb8e980e9138deef4ca3d1c3cb3f"} Aug 05 16:32:21.577110 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:21.577073 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vq474" event={"ID":"6d962e48-deee-46e7-b198-c8c29ab0f258","Type":"ContainerStarted","Data":"1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42"} Aug 05 16:32:21.578689 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:21.578663 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" event={"ID":"d8b8fa28-e50c-4a67-9f58-13df531c13c7","Type":"ContainerStarted","Data":"0e325aad1b33e644059bd55da3dcca08366edb2e7057b7db86700b959fed4d54"} Aug 05 16:32:21.578783 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:21.578757 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" Aug 05 16:32:21.594194 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:21.594144 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-vq474" podStartSLOduration=1.367003375 podStartE2EDuration="3.594129399s" totalImagesPullingTime="2.227126024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:32:21.592619704 +0000 UTC m=+564.951871549" watchObservedRunningTime="2026-08-05 16:32:21.594129399 +0000 UTC m=+564.953381222" Aug 05 16:32:21.611510 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:21.611449 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" podStartSLOduration=1.812062678 podStartE2EDuration="4.611427026s" totalImagesPullingTime="2.799364348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:32:21.60951572 +0000 UTC m=+564.968767566" watchObservedRunningTime="2026-08-05 16:32:21.611427026 +0000 UTC m=+564.970678874" Aug 05 16:32:32.585187 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:32.585153 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr" Aug 05 16:32:34.002989 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:34.002959 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:34.211416 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:34.211381 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:34.501441 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:34.501405 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:47.121959 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:47.121880 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:47.475987 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:47.475897 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:49.844257 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:49.844217 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vq474"] Aug 05 16:32:49.844775 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:49.844451 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-vq474" podUID="6d962e48-deee-46e7-b198-c8c29ab0f258" containerName="authorino" containerID="cri-o://1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42" gracePeriod=30 Aug 05 16:32:50.102292 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.102229 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vq474" Aug 05 16:32:50.150525 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.150485 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sqnl\" (UniqueName: \"kubernetes.io/projected/6d962e48-deee-46e7-b198-c8c29ab0f258-kube-api-access-5sqnl\") pod \"6d962e48-deee-46e7-b198-c8c29ab0f258\" (UID: \"6d962e48-deee-46e7-b198-c8c29ab0f258\") " Aug 05 16:32:50.158013 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.157832 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d962e48-deee-46e7-b198-c8c29ab0f258-kube-api-access-5sqnl" (OuterVolumeSpecName: "kube-api-access-5sqnl") pod "6d962e48-deee-46e7-b198-c8c29ab0f258" (UID: "6d962e48-deee-46e7-b198-c8c29ab0f258"). InnerVolumeSpecName "kube-api-access-5sqnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:32:50.252824 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.252749 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sqnl\" (UniqueName: \"kubernetes.io/projected/6d962e48-deee-46e7-b198-c8c29ab0f258-kube-api-access-5sqnl\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:32:50.723586 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.723548 2575 generic.go:356] "Generic (PLEG): container finished" podID="6d962e48-deee-46e7-b198-c8c29ab0f258" containerID="1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42" exitCode=0 Aug 05 16:32:50.723773 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.723608 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vq474" Aug 05 16:32:50.723773 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.723637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vq474" event={"ID":"6d962e48-deee-46e7-b198-c8c29ab0f258","Type":"ContainerDied","Data":"1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42"} Aug 05 16:32:50.723773 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.723683 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vq474" event={"ID":"6d962e48-deee-46e7-b198-c8c29ab0f258","Type":"ContainerDied","Data":"f7641868cb74bb6c0312643e9491d1de1c5aeb8e980e9138deef4ca3d1c3cb3f"} Aug 05 16:32:50.723773 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.723702 2575 scope.go:117] "RemoveContainer" containerID="1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42" Aug 05 16:32:50.732602 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.732582 2575 scope.go:117] "RemoveContainer" containerID="1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42" Aug 05 16:32:50.732956 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:32:50.732933 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42\": container with ID starting with 1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42 not found: ID does not exist" containerID="1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42" Aug 05 16:32:50.733064 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.732967 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42"} err="failed to get container status \"1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42\": rpc error: code = NotFound desc = could not find container \"1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42\": container with ID starting with 1befe337ea3bd947c1166b281e7cbd9a6e0ed7120551eb889bfcb48a5ab3bf42 not found: ID does not exist" Aug 05 16:32:50.745371 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.745336 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vq474"] Aug 05 16:32:50.749401 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:50.749374 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vq474"] Aug 05 16:32:51.120889 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:51.120857 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d962e48-deee-46e7-b198-c8c29ab0f258" path="/var/lib/kubelet/pods/6d962e48-deee-46e7-b198-c8c29ab0f258/volumes" Aug 05 16:32:54.874253 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:54.874225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:54.933600 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:54.933564 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:57.038158 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:57.038128 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:32:57.040657 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:57.040635 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:32:57.905028 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:57.904992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:58.001443 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:58.001402 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:59.776771 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:32:59.776727 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:05.635901 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:33:05.635865 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:16.736587 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:33:16.736554 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:16.793565 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:33:16.793529 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:29.535891 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:33:29.535858 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:29.634278 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:33:29.634242 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:31.035005 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:33:31.034966 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:32.293443 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:33:32.293406 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:32.513905 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:33:32.513870 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:33:42.135614 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:33:42.135582 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:42.224706 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:33:42.224674 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:34:21.069652 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:34:21.069620 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:34:21.446921 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:34:21.446841 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:34:24.975779 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:34:24.975740 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:34:25.444173 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:34:25.444138 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:34:41.350016 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:34:41.349987 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:34:41.404397 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:34:41.404368 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:34:43.993885 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:34:43.993849 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:34:44.942555 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:34:44.942527 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:34:53.513873 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:34:53.513756 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:35:02.069574 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:02.069538 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:35:03.252544 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:03.252509 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:35:11.852310 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:11.852276 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:35:19.583773 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:19.583735 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:35:21.239601 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.239565 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45"] Aug 05 16:35:21.240070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.239981 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d962e48-deee-46e7-b198-c8c29ab0f258" containerName="authorino" Aug 05 16:35:21.240070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.239993 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d962e48-deee-46e7-b198-c8c29ab0f258" containerName="authorino" Aug 05 16:35:21.240070 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.240057 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d962e48-deee-46e7-b198-c8c29ab0f258" containerName="authorino" Aug 05 16:35:21.243016 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.242998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.247391 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.247372 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 05 16:35:21.247536 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.247449 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 05 16:35:21.247536 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.247459 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 05 16:35:21.247536 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.247514 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g97pd\"" Aug 05 16:35:21.255161 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.255139 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45"] Aug 05 16:35:21.275562 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.275534 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:35:21.302403 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.302379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.302569 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.302437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30d24b56-d8cf-4c23-ac86-47a8e15be529-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.302569 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.302456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.302569 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.302505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.302569 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.302563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qtf4\" (UniqueName: \"kubernetes.io/projected/30d24b56-d8cf-4c23-ac86-47a8e15be529-kube-api-access-5qtf4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.302744 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.302599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.403708 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.403678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30d24b56-d8cf-4c23-ac86-47a8e15be529-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.403708 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.403708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.403954 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.403737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.403954 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.403856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qtf4\" (UniqueName: \"kubernetes.io/projected/30d24b56-d8cf-4c23-ac86-47a8e15be529-kube-api-access-5qtf4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.403954 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.403896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.403954 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.403926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.404155 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.404141 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.404214 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.404162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.404214 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.404189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.406288 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.406266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30d24b56-d8cf-4c23-ac86-47a8e15be529-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.406449 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.406431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30d24b56-d8cf-4c23-ac86-47a8e15be529-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.412902 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.412880 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qtf4\" (UniqueName: \"kubernetes.io/projected/30d24b56-d8cf-4c23-ac86-47a8e15be529-kube-api-access-5qtf4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wpx45\" (UID: \"30d24b56-d8cf-4c23-ac86-47a8e15be529\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.546913 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.546878 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:35:21.552452 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.552430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:21.684462 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.684441 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45"] Aug 05 16:35:21.686347 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:35:21.686318 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30d24b56_d8cf_4c23_ac86_47a8e15be529.slice/crio-12400cbe3f8e12107e499a29c35a40dcff4ad0e87eac9af6652b4b35526a2726 WatchSource:0}: Error finding container 12400cbe3f8e12107e499a29c35a40dcff4ad0e87eac9af6652b4b35526a2726: Status 404 returned error can't find the container with id 12400cbe3f8e12107e499a29c35a40dcff4ad0e87eac9af6652b4b35526a2726 Aug 05 16:35:21.687639 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:21.687621 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:35:22.432471 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:22.432436 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" event={"ID":"30d24b56-d8cf-4c23-ac86-47a8e15be529","Type":"ContainerStarted","Data":"12400cbe3f8e12107e499a29c35a40dcff4ad0e87eac9af6652b4b35526a2726"} Aug 05 16:35:23.438357 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:23.438327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" event={"ID":"30d24b56-d8cf-4c23-ac86-47a8e15be529","Type":"ContainerStarted","Data":"ac164f341df7029a6c1c6f36639240b8c862384e227cbce9c4581dba1214f1ae"} Aug 05 16:35:23.457975 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:23.457930 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" podStartSLOduration=1.33122126 podStartE2EDuration="2.45791577s" totalImagesPullingTime="1.12669451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:35:23.457449226 +0000 UTC m=+746.816701072" watchObservedRunningTime="2026-08-05 16:35:23.45791577 +0000 UTC m=+746.817167612" Aug 05 16:35:30.952141 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:30.952108 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:35:31.553174 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:31.553138 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:31.553174 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:31.553179 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:31.565620 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:31.565595 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:35:32.490198 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:35:32.490171 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wpx45" Aug 05 16:36:52.444164 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.444128 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bb787fc85-5hbbh"] Aug 05 16:36:52.447684 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.447662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bb787fc85-5hbbh" Aug 05 16:36:52.451787 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.451760 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 05 16:36:52.451787 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.451784 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mh5mq\"" Aug 05 16:36:52.457319 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.457298 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bb787fc85-5hbbh"] Aug 05 16:36:52.560363 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.560320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8dfe9c65-c1c9-42c6-9be0-5a25cbb26430-tls-cert\") pod \"authorino-6bb787fc85-5hbbh\" (UID: \"8dfe9c65-c1c9-42c6-9be0-5a25cbb26430\") " pod="kuadrant-system/authorino-6bb787fc85-5hbbh" Aug 05 16:36:52.560363 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.560371 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v264b\" (UniqueName: \"kubernetes.io/projected/8dfe9c65-c1c9-42c6-9be0-5a25cbb26430-kube-api-access-v264b\") pod \"authorino-6bb787fc85-5hbbh\" (UID: \"8dfe9c65-c1c9-42c6-9be0-5a25cbb26430\") " pod="kuadrant-system/authorino-6bb787fc85-5hbbh" Aug 05 16:36:52.661631 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.661590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8dfe9c65-c1c9-42c6-9be0-5a25cbb26430-tls-cert\") pod \"authorino-6bb787fc85-5hbbh\" (UID: \"8dfe9c65-c1c9-42c6-9be0-5a25cbb26430\") " pod="kuadrant-system/authorino-6bb787fc85-5hbbh" Aug 05 16:36:52.661631 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.661635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v264b\" (UniqueName: \"kubernetes.io/projected/8dfe9c65-c1c9-42c6-9be0-5a25cbb26430-kube-api-access-v264b\") pod \"authorino-6bb787fc85-5hbbh\" (UID: \"8dfe9c65-c1c9-42c6-9be0-5a25cbb26430\") " pod="kuadrant-system/authorino-6bb787fc85-5hbbh" Aug 05 16:36:52.664373 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.664335 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8dfe9c65-c1c9-42c6-9be0-5a25cbb26430-tls-cert\") pod \"authorino-6bb787fc85-5hbbh\" (UID: \"8dfe9c65-c1c9-42c6-9be0-5a25cbb26430\") " pod="kuadrant-system/authorino-6bb787fc85-5hbbh" Aug 05 16:36:52.671039 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.671012 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v264b\" (UniqueName: \"kubernetes.io/projected/8dfe9c65-c1c9-42c6-9be0-5a25cbb26430-kube-api-access-v264b\") pod \"authorino-6bb787fc85-5hbbh\" (UID: \"8dfe9c65-c1c9-42c6-9be0-5a25cbb26430\") " pod="kuadrant-system/authorino-6bb787fc85-5hbbh" Aug 05 16:36:52.757617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.757587 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bb787fc85-5hbbh" Aug 05 16:36:52.894261 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:52.894237 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bb787fc85-5hbbh"] Aug 05 16:36:52.896270 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:36:52.896240 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dfe9c65_c1c9_42c6_9be0_5a25cbb26430.slice/crio-0f388bdf97416a275c3f4bd336273bc4fe932ea4221a097f51892a6c60871d39 WatchSource:0}: Error finding container 0f388bdf97416a275c3f4bd336273bc4fe932ea4221a097f51892a6c60871d39: Status 404 returned error can't find the container with id 0f388bdf97416a275c3f4bd336273bc4fe932ea4221a097f51892a6c60871d39 Aug 05 16:36:53.871486 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:53.871449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bb787fc85-5hbbh" event={"ID":"8dfe9c65-c1c9-42c6-9be0-5a25cbb26430","Type":"ContainerStarted","Data":"81f253c15a9e73a5713914f93f5a251491211a87553d2139ed88a73dcbef7921"} Aug 05 16:36:53.871486 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:53.871491 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bb787fc85-5hbbh" event={"ID":"8dfe9c65-c1c9-42c6-9be0-5a25cbb26430","Type":"ContainerStarted","Data":"0f388bdf97416a275c3f4bd336273bc4fe932ea4221a097f51892a6c60871d39"} Aug 05 16:36:53.893940 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:53.893828 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6bb787fc85-5hbbh" podStartSLOduration=1.164028265 podStartE2EDuration="1.893782305s" totalImagesPullingTime="729.75404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:36:53.892130541 +0000 UTC m=+837.251382383" watchObservedRunningTime="2026-08-05 16:36:53.893782305 +0000 UTC m=+837.253034149" Aug 05 16:36:58.854805 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:58.854756 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:36:58.945548 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:36:58.945514 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:37:00.174639 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:37:00.174607 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:37:00.275334 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:37:00.275296 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:37:27.454817 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:37:27.454763 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:37:36.869844 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:37:36.869808 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:37:43.791847 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:37:43.791809 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:37:43.892940 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:37:43.892904 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:37:48.271319 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:37:48.271240 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:37:57.068555 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:37:57.068526 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:37:57.069049 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:37:57.068746 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:38:01.698687 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:38:01.698647 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:39:04.295928 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:39:04.295892 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:39:06.010613 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:39:06.010577 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:39:28.357112 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:39:28.357021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:39:38.559060 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:39:38.559029 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:40:08.336739 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:08.336701 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:40:08.434241 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:08.434204 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:40:08.557686 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:08.557653 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:40:14.557397 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:14.557360 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:40:14.675131 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:14.675098 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:40:14.771585 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:14.771547 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:40:14.946035 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:14.945948 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:40:18.112696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:18.112667 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:40:20.753499 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:20.753466 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:40:20.828482 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:20.828449 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:40:36.465139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:36.465098 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:40:43.646926 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:43.646889 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:40:59.053785 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:40:59.053704 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:41:05.056103 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:41:05.056070 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:41:14.544372 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:41:14.544333 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:41:20.247252 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:41:20.247214 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:41:34.950336 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:41:34.950297 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:41:40.252687 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:41:40.252653 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:41:45.444941 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:41:45.444900 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:41:48.881405 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:41:48.881327 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:42:07.053856 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:42:07.053822 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:42:17.743893 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:42:17.743861 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:42:40.852124 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:42:40.852087 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:42:51.259456 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:42:51.259417 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:42:56.549079 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:42:56.549037 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:42:57.106066 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:42:57.106037 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:42:57.106066 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:42:57.106049 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:43:04.761385 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:43:04.761346 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:43:11.950184 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:43:11.950150 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:43:16.857561 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:43:16.857527 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:43:22.250925 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:43:22.250891 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:43:30.162948 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:43:30.162911 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:43:42.059284 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:43:42.059255 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:43:48.942904 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:43:48.942826 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:43:59.755494 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:43:59.755463 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:44:04.958705 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:44:04.958666 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:44:11.289212 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:44:11.289170 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:44:19.854535 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:44:19.854500 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:44:24.759868 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:44:24.759829 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:44:35.966200 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:44:35.966164 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:45:30.727513 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:45:30.727437 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:45:32.250515 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:45:32.250485 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:46:52.450849 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:46:52.450771 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:46:59.456875 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:46:59.456842 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:47:12.654710 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:47:12.654679 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:47:13.375851 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:47:13.375819 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:47:21.670566 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:47:21.670527 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:47:29.840740 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:47:29.840704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:47:42.056973 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:47:42.056937 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:47:43.240546 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:47:43.240514 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:47:48.462033 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:47:48.461995 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:47:57.140691 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:47:57.140653 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:47:57.141155 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:47:57.140710 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:47:57.958278 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:47:57.958249 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:48:02.960631 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:48:02.960598 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:48:14.070884 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:48:14.070847 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:48:24.677713 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:48:24.677623 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:48:29.455552 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:48:29.455521 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:48:34.683696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:48:34.683655 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:48:44.559938 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:48:44.559903 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:48:54.753617 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:48:54.753578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:48:59.344995 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:48:59.344956 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:49:21.660423 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:21.660376 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:49:28.684543 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:28.684503 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:49:47.360081 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.360048 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll"] Aug 05 16:49:47.363515 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.363496 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.366418 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.366399 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f9594dd0-kserve-self-signed-certs\"" Aug 05 16:49:47.391302 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.391267 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll"] Aug 05 16:49:47.442978 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.442947 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:49:47.493301 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.493267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-home\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.493474 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.493356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-tmp-dir\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.493474 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.493383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb2edb3-a597-4416-b74c-b733806aa19a-tls-certs\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.493474 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.493406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbvc\" (UniqueName: \"kubernetes.io/projected/ebb2edb3-a597-4416-b74c-b733806aa19a-kube-api-access-gnbvc\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.493474 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.493438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-dshm\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.493474 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.493453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-model-cache\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.594115 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.594089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-home\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.594290 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.594127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-tmp-dir\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.594290 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.594144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb2edb3-a597-4416-b74c-b733806aa19a-tls-certs\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.594290 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.594162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbvc\" (UniqueName: \"kubernetes.io/projected/ebb2edb3-a597-4416-b74c-b733806aa19a-kube-api-access-gnbvc\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.594290 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.594185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-dshm\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.594290 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.594200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-model-cache\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.594610 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.594546 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-tmp-dir\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.594610 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.594546 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-home\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.594682 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.594663 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-model-cache\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.596473 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.596448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-dshm\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.596738 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.596719 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb2edb3-a597-4416-b74c-b733806aa19a-tls-certs\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.603877 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.603853 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbvc\" (UniqueName: \"kubernetes.io/projected/ebb2edb3-a597-4416-b74c-b733806aa19a-kube-api-access-gnbvc\") pod \"e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.672879 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.672776 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:47.804246 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:47.804220 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll"] Aug 05 16:49:47.806459 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:49:47.806423 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebb2edb3_a597_4416_b74c_b733806aa19a.slice/crio-26ca305e60694f3d489901bc85b59137e5275ce715ec557bfebd4583396ae90f WatchSource:0}: Error finding container 26ca305e60694f3d489901bc85b59137e5275ce715ec557bfebd4583396ae90f: Status 404 returned error can't find the container with id 26ca305e60694f3d489901bc85b59137e5275ce715ec557bfebd4583396ae90f Aug 05 16:49:48.413980 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:48.413946 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" event={"ID":"ebb2edb3-a597-4416-b74c-b733806aa19a","Type":"ContainerStarted","Data":"1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8"} Aug 05 16:49:48.413980 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:48.413986 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" event={"ID":"ebb2edb3-a597-4416-b74c-b733806aa19a","Type":"ContainerStarted","Data":"26ca305e60694f3d489901bc85b59137e5275ce715ec557bfebd4583396ae90f"} Aug 05 16:49:48.435777 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:48.435732 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" podStartSLOduration=1.4357163769999999 podStartE2EDuration="1.435716377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:49:48.4337312 +0000 UTC m=+1611.792983043" watchObservedRunningTime="2026-08-05 16:49:48.435716377 +0000 UTC m=+1611.794968221" Aug 05 16:49:49.757893 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:49.757857 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:49:57.673707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:57.673630 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:57.673707 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:57.673674 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:57.685996 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:57.685973 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:49:58.473849 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:49:58.473821 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:50:00.957834 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:00.957785 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf"] Aug 05 16:50:00.961294 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:00.961276 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:00.964345 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:00.964321 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f9594dd0-kserve-self-signed-certs\"" Aug 05 16:50:00.973620 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:00.973597 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf"] Aug 05 16:50:01.001448 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.001419 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:50:01.014595 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.014576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tmp-dir\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.014674 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.014606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2lm\" (UniqueName: \"kubernetes.io/projected/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-kube-api-access-fw2lm\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.014674 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.014628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-model-cache\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.014753 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.014689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tls-certs\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.014753 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.014705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-home\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.014753 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.014718 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-dshm\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.116050 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.116025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tmp-dir\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.116173 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.116074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2lm\" (UniqueName: \"kubernetes.io/projected/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-kube-api-access-fw2lm\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.116173 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.116101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-model-cache\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.116173 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.116163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tls-certs\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.116319 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.116184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-home\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.116319 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.116197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-dshm\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.116419 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.116401 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tmp-dir\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.116603 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.116579 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-home\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.116695 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.116610 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-model-cache\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.118615 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.118597 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-dshm\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.118989 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.118973 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tls-certs\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.129040 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.129023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2lm\" (UniqueName: \"kubernetes.io/projected/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-kube-api-access-fw2lm\") pod \"e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.270644 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.270618 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:01.429288 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.429263 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf"] Aug 05 16:50:01.430973 ip-10-0-128-17 kubenswrapper[2575]: W0805 16:50:01.430948 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5f1c5ce_64aa_4b0c_8f4c_f4ae4cad998c.slice/crio-cd59541991e9a37bcb63dbbc65efa3df593362feb6967dead5e298acd1a7dfb6 WatchSource:0}: Error finding container cd59541991e9a37bcb63dbbc65efa3df593362feb6967dead5e298acd1a7dfb6: Status 404 returned error can't find the container with id cd59541991e9a37bcb63dbbc65efa3df593362feb6967dead5e298acd1a7dfb6 Aug 05 16:50:01.478516 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:01.478485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" event={"ID":"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c","Type":"ContainerStarted","Data":"cd59541991e9a37bcb63dbbc65efa3df593362feb6967dead5e298acd1a7dfb6"} Aug 05 16:50:02.484719 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:02.484686 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" event={"ID":"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c","Type":"ContainerStarted","Data":"04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58"} Aug 05 16:50:02.503888 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:02.503841 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" podStartSLOduration=2.503826598 podStartE2EDuration="2.503826598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:50:02.50275648 +0000 UTC m=+1625.862008328" watchObservedRunningTime="2026-08-05 16:50:02.503826598 +0000 UTC m=+1625.863078442" Aug 05 16:50:03.559037 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:03.559005 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:50:11.271177 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:11.271140 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:11.271632 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:11.271193 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:11.283285 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:11.283261 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:11.538269 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:11.538187 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:28.127312 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:28.127279 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll"] Aug 05 16:50:28.171468 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:28.171436 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:50:29.778362 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:29.778329 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf"] Aug 05 16:50:29.800732 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:29.800706 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:50:30.057749 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:30.057665 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:50:39.856150 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:39.856117 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:50:43.146164 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.146128 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" podUID="ebb2edb3-a597-4416-b74c-b733806aa19a" containerName="main" containerID="cri-o://1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8" gracePeriod=45 Aug 05 16:50:43.386838 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.386816 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:50:43.495478 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.495450 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-dshm\") pod \"ebb2edb3-a597-4416-b74c-b733806aa19a\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " Aug 05 16:50:43.495605 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.495488 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-tmp-dir\") pod \"ebb2edb3-a597-4416-b74c-b733806aa19a\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " Aug 05 16:50:43.495605 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.495518 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-home\") pod \"ebb2edb3-a597-4416-b74c-b733806aa19a\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " Aug 05 16:50:43.495605 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.495570 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnbvc\" (UniqueName: \"kubernetes.io/projected/ebb2edb3-a597-4416-b74c-b733806aa19a-kube-api-access-gnbvc\") pod \"ebb2edb3-a597-4416-b74c-b733806aa19a\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " Aug 05 16:50:43.495766 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.495625 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb2edb3-a597-4416-b74c-b733806aa19a-tls-certs\") pod \"ebb2edb3-a597-4416-b74c-b733806aa19a\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " Aug 05 16:50:43.495766 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.495667 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-model-cache\") pod \"ebb2edb3-a597-4416-b74c-b733806aa19a\" (UID: \"ebb2edb3-a597-4416-b74c-b733806aa19a\") " Aug 05 16:50:43.495898 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.495752 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebb2edb3-a597-4416-b74c-b733806aa19a" (UID: "ebb2edb3-a597-4416-b74c-b733806aa19a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:50:43.495898 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.495838 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-home" (OuterVolumeSpecName: "home") pod "ebb2edb3-a597-4416-b74c-b733806aa19a" (UID: "ebb2edb3-a597-4416-b74c-b733806aa19a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:50:43.496020 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.495996 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebb2edb3-a597-4416-b74c-b733806aa19a" (UID: "ebb2edb3-a597-4416-b74c-b733806aa19a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:50:43.496078 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.496007 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:43.496078 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.496049 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:43.497871 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.497842 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-dshm" (OuterVolumeSpecName: "dshm") pod "ebb2edb3-a597-4416-b74c-b733806aa19a" (UID: "ebb2edb3-a597-4416-b74c-b733806aa19a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:50:43.497987 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.497906 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb2edb3-a597-4416-b74c-b733806aa19a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebb2edb3-a597-4416-b74c-b733806aa19a" (UID: "ebb2edb3-a597-4416-b74c-b733806aa19a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:50:43.497987 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.497928 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb2edb3-a597-4416-b74c-b733806aa19a-kube-api-access-gnbvc" (OuterVolumeSpecName: "kube-api-access-gnbvc") pod "ebb2edb3-a597-4416-b74c-b733806aa19a" (UID: "ebb2edb3-a597-4416-b74c-b733806aa19a"). InnerVolumeSpecName "kube-api-access-gnbvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:50:43.597388 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.597351 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbvc\" (UniqueName: \"kubernetes.io/projected/ebb2edb3-a597-4416-b74c-b733806aa19a-kube-api-access-gnbvc\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:43.597388 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.597379 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb2edb3-a597-4416-b74c-b733806aa19a-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:43.597388 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.597394 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:43.597604 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.597405 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebb2edb3-a597-4416-b74c-b733806aa19a-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:43.676890 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.676858 2575 generic.go:356] "Generic (PLEG): container finished" podID="ebb2edb3-a597-4416-b74c-b733806aa19a" containerID="1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8" exitCode=0 Aug 05 16:50:43.677038 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.676915 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" Aug 05 16:50:43.677038 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.676942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" event={"ID":"ebb2edb3-a597-4416-b74c-b733806aa19a","Type":"ContainerDied","Data":"1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8"} Aug 05 16:50:43.677038 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.676987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll" event={"ID":"ebb2edb3-a597-4416-b74c-b733806aa19a","Type":"ContainerDied","Data":"26ca305e60694f3d489901bc85b59137e5275ce715ec557bfebd4583396ae90f"} Aug 05 16:50:43.677038 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.677006 2575 scope.go:117] "RemoveContainer" containerID="1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8" Aug 05 16:50:43.686147 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.686127 2575 scope.go:117] "RemoveContainer" containerID="1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8" Aug 05 16:50:43.686389 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:50:43.686368 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8\": container with ID starting with 1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8 not found: ID does not exist" containerID="1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8" Aug 05 16:50:43.686430 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.686399 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8"} err="failed to get container status \"1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8\": rpc error: code = NotFound desc = could not find container \"1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8\": container with ID starting with 1b0510c4587822239e1371ba082a1bb52637edfb2208b7ad7a6d55cf0c942bb8 not found: ID does not exist" Aug 05 16:50:43.698018 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.697997 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll"] Aug 05 16:50:43.701989 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:43.701969 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f9594dd0-kserve-7677c4c975-6m9ll"] Aug 05 16:50:44.797083 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:44.797034 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" podUID="d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" containerName="main" containerID="cri-o://04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58" gracePeriod=45 Aug 05 16:50:44.955287 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:44.955249 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:50:45.041192 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.041166 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:45.111057 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.110981 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tls-certs\") pod \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " Aug 05 16:50:45.111057 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111028 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-home\") pod \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " Aug 05 16:50:45.111057 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111055 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-dshm\") pod \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " Aug 05 16:50:45.111327 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111093 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw2lm\" (UniqueName: \"kubernetes.io/projected/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-kube-api-access-fw2lm\") pod \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " Aug 05 16:50:45.111327 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111144 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-model-cache\") pod \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " Aug 05 16:50:45.111327 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111283 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tmp-dir\") pod \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\" (UID: \"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c\") " Aug 05 16:50:45.111489 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111313 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-home" (OuterVolumeSpecName: "home") pod "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" (UID: "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:50:45.111489 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111346 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" (UID: "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:50:45.111611 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111505 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" (UID: "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:50:45.111611 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111581 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:45.111611 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111601 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:45.111949 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.111613 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:45.113358 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.113332 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-dshm" (OuterVolumeSpecName: "dshm") pod "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" (UID: "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:50:45.113358 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.113332 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" (UID: "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:50:45.113497 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.113362 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-kube-api-access-fw2lm" (OuterVolumeSpecName: "kube-api-access-fw2lm") pod "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" (UID: "d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c"). InnerVolumeSpecName "kube-api-access-fw2lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:50:45.121540 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.121515 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb2edb3-a597-4416-b74c-b733806aa19a" path="/var/lib/kubelet/pods/ebb2edb3-a597-4416-b74c-b733806aa19a/volumes" Aug 05 16:50:45.212335 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.212311 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:45.212335 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.212335 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:45.212509 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.212348 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2lm\" (UniqueName: \"kubernetes.io/projected/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c-kube-api-access-fw2lm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 16:50:45.689627 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.689593 2575 generic.go:356] "Generic (PLEG): container finished" podID="d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" containerID="04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58" exitCode=0 Aug 05 16:50:45.689627 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.689629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" event={"ID":"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c","Type":"ContainerDied","Data":"04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58"} Aug 05 16:50:45.689909 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.689652 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" Aug 05 16:50:45.689909 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.689666 2575 scope.go:117] "RemoveContainer" containerID="04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58" Aug 05 16:50:45.689909 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.689656 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf" event={"ID":"d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c","Type":"ContainerDied","Data":"cd59541991e9a37bcb63dbbc65efa3df593362feb6967dead5e298acd1a7dfb6"} Aug 05 16:50:45.698069 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.698050 2575 scope.go:117] "RemoveContainer" containerID="04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58" Aug 05 16:50:45.698326 ip-10-0-128-17 kubenswrapper[2575]: E0805 16:50:45.698308 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58\": container with ID starting with 04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58 not found: ID does not exist" containerID="04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58" Aug 05 16:50:45.698382 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.698339 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58"} err="failed to get container status \"04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58\": rpc error: code = NotFound desc = could not find container \"04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58\": container with ID starting with 04aab19b4d0ab2ec2e2fde9d2bd5718e3d1285198add55e5d27970ff71320b58 not found: ID does not exist" Aug 05 16:50:45.708233 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.708207 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf"] Aug 05 16:50:45.712518 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:45.712498 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f9594dd0-kserve-64b649cc4c-9fjbf"] Aug 05 16:50:47.119641 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:47.119609 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" path="/var/lib/kubelet/pods/d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c/volumes" Aug 05 16:50:51.451232 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:50:51.451204 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:51:02.443408 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:51:02.443373 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:51:08.544528 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:51:08.544491 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:51:16.257044 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:51:16.257016 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:51:24.562042 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:51:24.561963 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:51:34.770962 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:51:34.770924 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:51:41.566755 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:51:41.566722 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:51:57.459932 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:51:57.459898 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:52:04.157438 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:04.157400 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:52:15.750429 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:15.750393 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:52:22.363552 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:22.363520 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:52:26.680697 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:26.680668 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:52:31.463809 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:31.463757 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:52:36.758851 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:36.758815 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:52:42.678549 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:42.678516 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:52:42.776307 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:42.776286 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:52:44.859450 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:44.859410 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:52:50.849543 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:50.849452 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:52:57.167655 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:57.167626 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:52:57.168171 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:57.168155 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:52:57.367141 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:57.367099 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:52:59.259613 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:52:59.259581 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:07.858054 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:53:07.858019 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:53:14.361971 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:53:14.361944 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:53:25.156833 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:53:25.156783 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:53:40.353034 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:53:40.352999 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:53:46.753977 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:53:46.753937 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:53:50.102438 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:53:50.102408 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:53.353871 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:53:53.353837 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:54:04.462177 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:04.462137 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:54:12.857150 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:12.857096 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:54:16.142812 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:16.142772 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:18.464170 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:18.464090 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:54:19.327361 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:19.327324 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:19.434013 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:19.433974 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:19.803297 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:19.803263 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:25.752179 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:25.752147 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:26.770029 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:26.769993 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:26.869144 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:26.869115 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:29.119951 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:29.119922 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:29.218981 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:29.218957 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:30.735122 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:30.735080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:30.986832 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:30.986733 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:31.238096 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:31.238018 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:31.340414 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:31.340382 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:33.272487 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:33.272457 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:54:49.850764 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:49.850734 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:49.935247 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:49.935216 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:50.008956 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:50.008929 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:50.094607 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:50.094577 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:50.390167 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:50.390142 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:50.618215 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:50.618184 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:50.938319 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:50.938290 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:51.026951 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:51.026923 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:51.135251 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:51.135226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:51.489239 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:51.489209 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:52.085396 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:52.085366 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:53.162807 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:53.162748 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:54:54.361660 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:54.361631 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:54.630267 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:54.630183 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:54.954297 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:54.954202 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:55.189675 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:55.189647 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:55.287776 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:54:55.287751 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:03.474417 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:03.474385 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:55:25.439584 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:25.439555 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:25.527553 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:25.527521 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:25.618348 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:25.618313 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:25.695823 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:25.695733 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:25.892258 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:25.892225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:26.123614 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:26.123590 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:26.370242 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:26.370209 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:26.732397 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:26.732371 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:26.839113 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:26.839083 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:28.664215 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:28.664177 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:55:29.508012 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:29.507977 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:29.604560 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:29.604531 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:30.648588 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:30.648551 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:30.812250 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:30.812214 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:31.335217 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:31.335174 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:31.504773 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:31.504742 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:38.163482 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:55:38.163452 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:56:01.324575 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:01.324503 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:01.438705 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:01.438666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:01.535392 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:01.535364 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:01.618037 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:01.617951 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:02.828431 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:02.828393 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:03.079711 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:03.079629 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:03.332087 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:03.331990 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:03.709342 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:03.709263 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:03.810187 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:03.810161 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:04.455083 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:04.455052 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:56:05.738507 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:05.738472 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:05.821445 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:05.821417 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:06.969035 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:06.968993 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:07.131044 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:07.131013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:07.665445 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:07.665413 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:07.870762 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:07.870732 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:14.462398 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:14.462367 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:56:37.653832 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:37.653788 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:37.741292 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:37.741267 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:37.832005 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:37.831966 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:37.935637 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:37.935550 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:38.100139 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:38.100111 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:38.331782 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:38.331754 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:38.852634 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:38.852586 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:38.946782 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:38.946727 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:39.048299 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:39.048269 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:40.791175 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:40.791148 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:56:50.857226 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:50.857196 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:52.267180 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:52.267143 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:56:57.897476 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:57.897443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:58.019400 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:58.019370 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:58.130373 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:58.130340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:58.228280 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:58.228210 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:58.690163 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:56:58.690125 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:24.688083 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:57:24.687997 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:31.441285 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:57:31.441248 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:32.964010 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:57:32.963978 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:47.592134 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:57:47.592099 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:51.528375 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:57:51.528343 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:53.810495 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:57:53.810443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:53.937251 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:57:53.937223 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:54.027545 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:57:54.027513 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:57.200329 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:57:57.200300 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:57:57.202560 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:57:57.202542 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 16:58:04.626103 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:04.626060 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:15.693765 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:15.693723 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:22.145977 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:22.145929 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:22.248696 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:22.248661 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:22.346486 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:22.346433 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:22.446711 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:22.446630 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:23.120716 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:23.120683 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:23.224665 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:23.224637 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:23.299874 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:23.299848 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:23.571819 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:23.571769 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:58:24.973083 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:24.973045 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:32.301912 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:32.301881 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:32.410198 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:32.410168 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:33.064640 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:33.064605 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:34.090018 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:34.089984 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:58:40.237312 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:40.237270 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:40.384747 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:40.384716 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:40.567721 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:40.567657 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:40.914384 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:40.914302 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:41.007701 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:41.007667 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:44.289776 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:44.289740 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:44.390145 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:44.390109 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:45.687759 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:45.687724 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:45.962920 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:45.962842 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:46.057741 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:46.057702 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:46.167906 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:46.167864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:48.453108 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:58:48.453031 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:59:09.283995 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:09.283954 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:10.049356 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:10.049322 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:10.150920 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:10.150877 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:10.486176 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:10.486092 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:16.601996 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:16.601961 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:16.673135 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:16.673101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:26.974264 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:26.974227 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:27.075751 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:27.075714 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:27.607013 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:27.606979 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:27.702487 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:27.702455 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:27.773026 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:27.772991 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:27.844293 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:27.844243 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:28.243656 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:28.243621 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:29.752603 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:29.752565 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 16:59:34.463245 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:34.463181 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:34.564444 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:34.564403 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:34.757784 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:34.757748 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:35.008158 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:35.008034 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:35.200217 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:35.200165 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:35.281776 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:35.281691 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:35.382750 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:35.382720 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:41.465802 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:41.465768 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:41.655526 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:41.655493 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:41.843009 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:41.842981 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:41.980525 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:41.980467 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:47.956195 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:47.956159 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:48.056818 ip-10-0-128-17 kubenswrapper[2575]: I0805 16:59:48.056758 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:00:07.554443 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:00:07.554403 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:00:39.689931 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:00:39.689893 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:00:39.790197 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:00:39.790146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:11.880632 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:11.880595 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:21.666498 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:21.666466 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:21.767443 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:21.767396 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:51.549578 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:51.549537 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:01:51.778351 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:51.778295 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:52.162308 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:52.162247 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:52.528642 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:52.528591 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:56.365192 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:56.365157 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:56.464663 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:56.464623 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:56.562385 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:56.562353 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:57.073652 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:57.073614 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:57.171368 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:57.171325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:59.550163 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:01:59.550128 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:02:20.437006 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:20.436944 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:33.154078 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.154045 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh"] Aug 05 17:02:33.154590 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.154393 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb2edb3-a597-4416-b74c-b733806aa19a" containerName="main" Aug 05 17:02:33.154590 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.154403 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb2edb3-a597-4416-b74c-b733806aa19a" containerName="main" Aug 05 17:02:33.154590 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.154418 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" containerName="main" Aug 05 17:02:33.154590 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.154423 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" containerName="main" Aug 05 17:02:33.154590 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.154477 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f1c5ce-64aa-4b0c-8f4c-f4ae4cad998c" containerName="main" Aug 05 17:02:33.154590 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.154487 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb2edb3-a597-4416-b74c-b733806aa19a" containerName="main" Aug 05 17:02:33.157182 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.157165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.160172 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.160147 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-febb9564\"/\"kube-root-ca.crt\"" Aug 05 17:02:33.161508 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.161490 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-febb9564\"/\"default-dockercfg-pzkzz\"" Aug 05 17:02:33.161629 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.161552 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-febb9564\"/\"openshift-service-ca.crt\"" Aug 05 17:02:33.161697 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.161650 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-febb9564\"/\"e2e-lifecycle-model-febb9564-kserve-self-signed-certs\"" Aug 05 17:02:33.167261 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.167232 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh"] Aug 05 17:02:33.221142 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.221110 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-dshm\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.221333 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.221165 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-model-cache\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.221333 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.221188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tmp-dir\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.221449 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.221338 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-home\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.221449 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.221371 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tls-certs\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.221449 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.221427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-299dj\" (UniqueName: \"kubernetes.io/projected/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-kube-api-access-299dj\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.322423 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.322392 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-dshm\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.322579 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.322436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-model-cache\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.322579 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.322464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tmp-dir\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.322579 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.322497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-home\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.322579 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.322517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tls-certs\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.322579 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.322553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-299dj\" (UniqueName: \"kubernetes.io/projected/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-kube-api-access-299dj\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.322883 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.322859 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-model-cache\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.322947 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.322894 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tmp-dir\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.323002 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.322988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-home\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.324848 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.324821 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-dshm\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.325097 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.325081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tls-certs\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.332024 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.332005 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-299dj\" (UniqueName: \"kubernetes.io/projected/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-kube-api-access-299dj\") pod \"e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.468288 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.468202 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:33.478432 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.478406 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:33.603762 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:33.603687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh"] Aug 05 17:02:33.606380 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:02:33.606348 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29aee7bd_5f5d_4aab_b328_9506b2e4b27a.slice/crio-2d3d051391e1b247928362ce7111a743ccd89c975ad69344205e6a8ddfd645b7 WatchSource:0}: Error finding container 2d3d051391e1b247928362ce7111a743ccd89c975ad69344205e6a8ddfd645b7: Status 404 returned error can't find the container with id 2d3d051391e1b247928362ce7111a743ccd89c975ad69344205e6a8ddfd645b7 Aug 05 17:02:34.008773 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:34.008741 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" event={"ID":"29aee7bd-5f5d-4aab-b328-9506b2e4b27a","Type":"ContainerStarted","Data":"fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602"} Aug 05 17:02:34.008773 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:34.008781 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" event={"ID":"29aee7bd-5f5d-4aab-b328-9506b2e4b27a","Type":"ContainerStarted","Data":"2d3d051391e1b247928362ce7111a743ccd89c975ad69344205e6a8ddfd645b7"} Aug 05 17:02:34.028626 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:34.028567 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" podStartSLOduration=1.028549735 podStartE2EDuration="1.028549735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:02:34.026897541 +0000 UTC m=+2377.386149385" watchObservedRunningTime="2026-08-05 17:02:34.028549735 +0000 UTC m=+2377.387801557" Aug 05 17:02:35.727165 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:35.727132 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:35.850312 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:35.850281 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:02:43.468922 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:43.468867 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:43.468922 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:43.468929 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:43.481260 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:43.481231 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:44.067538 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:44.067508 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:02:46.482899 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:46.482858 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:51.654834 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:51.654780 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:02:57.228683 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:57.228647 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 17:02:57.232137 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:02:57.232114 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 17:03:14.150938 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:14.150900 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:03:16.436389 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:16.436332 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:16.511812 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:16.511732 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:16.612013 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:16.611981 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:16.680297 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:16.680267 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:19.899557 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:19.899475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:21.848335 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:21.848305 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:03:22.522215 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:22.522183 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:24.881852 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:24.881812 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh"] Aug 05 17:03:24.978507 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:24.978472 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:39.901859 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:39.901787 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" podUID="29aee7bd-5f5d-4aab-b328-9506b2e4b27a" containerName="main" containerID="cri-o://fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602" gracePeriod=45 Aug 05 17:03:40.141913 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.141889 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:03:40.230719 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.230633 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-dshm\") pod \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " Aug 05 17:03:40.230719 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.230668 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-model-cache\") pod \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " Aug 05 17:03:40.230719 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.230692 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-299dj\" (UniqueName: \"kubernetes.io/projected/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-kube-api-access-299dj\") pod \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " Aug 05 17:03:40.231038 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.230736 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tmp-dir\") pod \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " Aug 05 17:03:40.231038 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.230751 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-home\") pod \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " Aug 05 17:03:40.231038 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.230765 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tls-certs\") pod \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\" (UID: \"29aee7bd-5f5d-4aab-b328-9506b2e4b27a\") " Aug 05 17:03:40.231038 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.230992 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-model-cache" (OuterVolumeSpecName: "model-cache") pod "29aee7bd-5f5d-4aab-b328-9506b2e4b27a" (UID: "29aee7bd-5f5d-4aab-b328-9506b2e4b27a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:03:40.231243 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.231050 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29aee7bd-5f5d-4aab-b328-9506b2e4b27a" (UID: "29aee7bd-5f5d-4aab-b328-9506b2e4b27a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:03:40.231243 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.231106 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-home" (OuterVolumeSpecName: "home") pod "29aee7bd-5f5d-4aab-b328-9506b2e4b27a" (UID: "29aee7bd-5f5d-4aab-b328-9506b2e4b27a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:03:40.231243 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.231209 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:03:40.231366 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.231253 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:03:40.231366 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.231267 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:03:40.233299 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.233270 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-dshm" (OuterVolumeSpecName: "dshm") pod "29aee7bd-5f5d-4aab-b328-9506b2e4b27a" (UID: "29aee7bd-5f5d-4aab-b328-9506b2e4b27a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:03:40.233299 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.233272 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29aee7bd-5f5d-4aab-b328-9506b2e4b27a" (UID: "29aee7bd-5f5d-4aab-b328-9506b2e4b27a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:03:40.233457 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.233296 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-kube-api-access-299dj" (OuterVolumeSpecName: "kube-api-access-299dj") pod "29aee7bd-5f5d-4aab-b328-9506b2e4b27a" (UID: "29aee7bd-5f5d-4aab-b328-9506b2e4b27a"). InnerVolumeSpecName "kube-api-access-299dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:03:40.321068 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.321033 2575 generic.go:356] "Generic (PLEG): container finished" podID="29aee7bd-5f5d-4aab-b328-9506b2e4b27a" containerID="fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602" exitCode=0 Aug 05 17:03:40.321263 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.321076 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" Aug 05 17:03:40.321263 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.321120 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" event={"ID":"29aee7bd-5f5d-4aab-b328-9506b2e4b27a","Type":"ContainerDied","Data":"fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602"} Aug 05 17:03:40.321263 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.321170 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh" event={"ID":"29aee7bd-5f5d-4aab-b328-9506b2e4b27a","Type":"ContainerDied","Data":"2d3d051391e1b247928362ce7111a743ccd89c975ad69344205e6a8ddfd645b7"} Aug 05 17:03:40.321263 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.321192 2575 scope.go:117] "RemoveContainer" containerID="fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602" Aug 05 17:03:40.330359 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.330340 2575 scope.go:117] "RemoveContainer" containerID="fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602" Aug 05 17:03:40.330605 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:03:40.330589 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602\": container with ID starting with fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602 not found: ID does not exist" containerID="fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602" Aug 05 17:03:40.330649 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.330615 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602"} err="failed to get container status \"fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602\": rpc error: code = NotFound desc = could not find container \"fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602\": container with ID starting with fd2addfdd3380f1d4683680e701df3ff4a9f759cae49c00a32d1a5f92bfe7602 not found: ID does not exist" Aug 05 17:03:40.332051 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.332033 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:03:40.332104 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.332054 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-299dj\" (UniqueName: \"kubernetes.io/projected/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-kube-api-access-299dj\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:03:40.332104 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.332064 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29aee7bd-5f5d-4aab-b328-9506b2e4b27a-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:03:40.343600 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.343569 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh"] Aug 05 17:03:40.349174 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:40.349149 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-febb9564/e2e-lifecycle-model-febb9564-kserve-86dd96fd4c-9nqgh"] Aug 05 17:03:41.120008 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:41.119976 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29aee7bd-5f5d-4aab-b328-9506b2e4b27a" path="/var/lib/kubelet/pods/29aee7bd-5f5d-4aab-b328-9506b2e4b27a/volumes" Aug 05 17:03:56.492984 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:56.492925 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:57.076684 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:03:57.076641 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:02.668843 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:02.668781 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:02.770256 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:02.770224 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:03.033817 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:03.033769 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:03.230080 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:03.230049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:05.056725 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:05.056683 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:04:14.595473 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:14.595435 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:14.696628 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:14.696593 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:14.793122 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:14.793094 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:15.509986 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:15.509939 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:15.606995 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:15.606966 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:15.709439 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:15.709408 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:17.003284 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:17.003254 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:17.245359 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:17.245319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:04:23.591535 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:23.591506 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:23.662491 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:23.662462 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:23.948957 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:23.948880 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:04:24.273871 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:24.273843 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:30.545429 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:30.545391 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:30.608373 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:30.608319 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:32.657459 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:32.657427 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:04:41.104946 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:41.104906 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:41.343192 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:04:41.343149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:02.423158 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:02.423090 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:08.490181 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:08.490143 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:08.590379 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:08.590336 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:09.138509 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:09.138454 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:09.215042 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:09.214965 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:09.651370 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:09.651336 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:12.324205 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:12.324167 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:12.423341 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:12.423309 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:13.594552 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:13.594520 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:13.836680 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:13.836648 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:14.206161 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:14.206130 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:14.304846 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:14.304781 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:16.543128 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:16.543091 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:05:37.405061 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:37.405029 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:37.766295 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:37.766268 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:38.116591 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:38.116494 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:38.446336 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:38.446236 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:41.514733 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:41.514700 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:41.609927 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:41.609890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:42.815634 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:42.815600 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:42.940267 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:42.940233 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:43.328648 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:43.328614 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:43.430200 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:43.430165 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:45.750335 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:05:45.750300 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:06:11.734426 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:11.734389 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:06:20.460685 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.460596 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx"] Aug 05 17:06:20.461178 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.461158 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29aee7bd-5f5d-4aab-b328-9506b2e4b27a" containerName="main" Aug 05 17:06:20.461254 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.461182 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="29aee7bd-5f5d-4aab-b328-9506b2e4b27a" containerName="main" Aug 05 17:06:20.461306 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.461283 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="29aee7bd-5f5d-4aab-b328-9506b2e4b27a" containerName="main" Aug 05 17:06:20.464251 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.464230 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.467526 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.467503 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-deb398\"/\"kube-root-ca.crt\"" Aug 05 17:06:20.468568 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.468550 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-deb398\"/\"default-dockercfg-mwmg7\"" Aug 05 17:06:20.468568 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.468560 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-deb398\"/\"openshift-service-ca.crt\"" Aug 05 17:06:20.468762 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.468560 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-deb398\"/\"test-model-deb398-kserve-self-signed-certs\"" Aug 05 17:06:20.473755 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.473735 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx"] Aug 05 17:06:20.494469 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.494440 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:06:20.614648 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.614619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-home\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.614767 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.614657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-dshm\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.614767 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.614698 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-model-cache\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.614767 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.614723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tmp-dir\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.614966 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.614786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tls-certs\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.614966 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.614851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5nv\" (UniqueName: \"kubernetes.io/projected/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-kube-api-access-rc5nv\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.715968 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.715893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-dshm\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.715968 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.715940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-model-cache\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.715968 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.715966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tmp-dir\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.716216 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.716008 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tls-certs\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.716216 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.716038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5nv\" (UniqueName: \"kubernetes.io/projected/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-kube-api-access-rc5nv\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.716216 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.716122 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-home\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.716449 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.716427 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tmp-dir\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.716501 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.716444 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-model-cache\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.716533 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.716506 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-home\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.718375 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.718355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-dshm\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.718645 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.718628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tls-certs\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.726011 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.725959 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5nv\" (UniqueName: \"kubernetes.io/projected/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-kube-api-access-rc5nv\") pod \"test-model-deb398-kserve-55bdc9d55f-x69sx\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.774803 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.774769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:20.908460 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:20.908387 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx"] Aug 05 17:06:20.910744 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:06:20.910714 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b7fc85d_deb8_49fd_b4c4_95f4aa822825.slice/crio-da69bacfb57270c81afddc8f2ef0f072067e848034d3f6d52bfb79f3b9505865 WatchSource:0}: Error finding container da69bacfb57270c81afddc8f2ef0f072067e848034d3f6d52bfb79f3b9505865: Status 404 returned error can't find the container with id da69bacfb57270c81afddc8f2ef0f072067e848034d3f6d52bfb79f3b9505865 Aug 05 17:06:21.091154 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:21.091122 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" event={"ID":"8b7fc85d-deb8-49fd-b4c4-95f4aa822825","Type":"ContainerStarted","Data":"0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db"} Aug 05 17:06:21.091154 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:21.091160 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" event={"ID":"8b7fc85d-deb8-49fd-b4c4-95f4aa822825","Type":"ContainerStarted","Data":"da69bacfb57270c81afddc8f2ef0f072067e848034d3f6d52bfb79f3b9505865"} Aug 05 17:06:21.109763 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:21.109715 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" podStartSLOduration=1.10970024 podStartE2EDuration="1.10970024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:06:21.107239076 +0000 UTC m=+2604.466490940" watchObservedRunningTime="2026-08-05 17:06:21.10970024 +0000 UTC m=+2604.468952085" Aug 05 17:06:22.988730 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:22.988703 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:06:30.455412 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:30.455372 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:06:30.775728 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:30.775688 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:30.775949 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:30.775744 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:30.788089 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:30.788058 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:31.151005 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:31.150927 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:06:46.453518 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.453478 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8"] Aug 05 17:06:46.455992 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.455974 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.460154 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.460124 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3602fc\"/\"default-dockercfg-75b6l\"" Aug 05 17:06:46.460317 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.460196 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3602fc\"/\"kube-root-ca.crt\"" Aug 05 17:06:46.460317 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.460138 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3602fc\"/\"openshift-service-ca.crt\"" Aug 05 17:06:46.460451 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.460332 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3602fc\"/\"test-model-3602fc-kserve-self-signed-certs\"" Aug 05 17:06:46.466383 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.466361 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8"] Aug 05 17:06:46.495921 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.495888 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:06:46.554545 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.554501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db64de6-c670-4ebf-9b97-abfc2656639a-tls-certs\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.554710 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.554558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m6fw\" (UniqueName: \"kubernetes.io/projected/9db64de6-c670-4ebf-9b97-abfc2656639a-kube-api-access-7m6fw\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.554710 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.554626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-tmp-dir\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.554850 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.554743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-home\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.554850 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.554775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-dshm\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.554955 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.554854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-model-cache\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.656282 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.656242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db64de6-c670-4ebf-9b97-abfc2656639a-tls-certs\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.656282 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.656293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6fw\" (UniqueName: \"kubernetes.io/projected/9db64de6-c670-4ebf-9b97-abfc2656639a-kube-api-access-7m6fw\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.656535 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.656315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-tmp-dir\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.656535 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.656392 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-home\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.656535 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.656423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-dshm\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.656535 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.656463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-model-cache\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.656873 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.656842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-tmp-dir\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.657008 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.656905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-home\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.657008 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.656954 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-model-cache\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.659009 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.658988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-dshm\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.659291 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.659271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db64de6-c670-4ebf-9b97-abfc2656639a-tls-certs\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.666025 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.665995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m6fw\" (UniqueName: \"kubernetes.io/projected/9db64de6-c670-4ebf-9b97-abfc2656639a-kube-api-access-7m6fw\") pod \"test-model-3602fc-kserve-896db9689-69bw8\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.767354 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.767307 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:46.912380 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:46.912352 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8"] Aug 05 17:06:46.914744 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:06:46.914719 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9db64de6_c670_4ebf_9b97_abfc2656639a.slice/crio-d3e33f1ef03b31ee0ac93217245051c1b89b043748dfd280c33e84795a2db271 WatchSource:0}: Error finding container d3e33f1ef03b31ee0ac93217245051c1b89b043748dfd280c33e84795a2db271: Status 404 returned error can't find the container with id d3e33f1ef03b31ee0ac93217245051c1b89b043748dfd280c33e84795a2db271 Aug 05 17:06:47.217648 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:47.217548 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" event={"ID":"9db64de6-c670-4ebf-9b97-abfc2656639a","Type":"ContainerStarted","Data":"fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1"} Aug 05 17:06:47.217648 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:47.217600 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" event={"ID":"9db64de6-c670-4ebf-9b97-abfc2656639a","Type":"ContainerStarted","Data":"d3e33f1ef03b31ee0ac93217245051c1b89b043748dfd280c33e84795a2db271"} Aug 05 17:06:47.234965 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:47.234909 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" podStartSLOduration=1.234889657 podStartE2EDuration="1.234889657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:06:47.234540667 +0000 UTC m=+2630.593792514" watchObservedRunningTime="2026-08-05 17:06:47.234889657 +0000 UTC m=+2630.594141502" Aug 05 17:06:49.547135 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:49.547093 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:06:56.767758 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:56.767707 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:56.768279 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:56.768002 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:56.780692 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:56.780663 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:57.281023 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:57.280990 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:06:58.344779 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:06:58.344744 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:07:30.618364 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:30.618329 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:31.639995 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:31.639955 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx"] Aug 05 17:07:31.661566 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:31.661543 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:31.854672 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:31.854640 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:31.928485 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:31.928421 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:32.568610 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:32.568573 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8"] Aug 05 17:07:32.581031 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:32.581001 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:36.850853 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:36.850821 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:07:43.407402 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:43.407372 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:45.531189 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:45.531155 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:45.546173 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:45.546144 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:07:46.659755 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:46.659695 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" podUID="8b7fc85d-deb8-49fd-b4c4-95f4aa822825" containerName="main" containerID="cri-o://0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db" gracePeriod=45 Aug 05 17:07:46.910674 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:46.910621 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:07:47.004164 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004143 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-model-cache\") pod \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " Aug 05 17:07:47.004289 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004177 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tls-certs\") pod \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " Aug 05 17:07:47.004289 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004205 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tmp-dir\") pod \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " Aug 05 17:07:47.004289 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004240 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-home\") pod \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " Aug 05 17:07:47.004289 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004277 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc5nv\" (UniqueName: \"kubernetes.io/projected/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-kube-api-access-rc5nv\") pod \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " Aug 05 17:07:47.004456 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-dshm\") pod \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\" (UID: \"8b7fc85d-deb8-49fd-b4c4-95f4aa822825\") " Aug 05 17:07:47.004456 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004407 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b7fc85d-deb8-49fd-b4c4-95f4aa822825" (UID: "8b7fc85d-deb8-49fd-b4c4-95f4aa822825"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:47.004456 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004440 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b7fc85d-deb8-49fd-b4c4-95f4aa822825" (UID: "8b7fc85d-deb8-49fd-b4c4-95f4aa822825"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:47.004613 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004536 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-home" (OuterVolumeSpecName: "home") pod "8b7fc85d-deb8-49fd-b4c4-95f4aa822825" (UID: "8b7fc85d-deb8-49fd-b4c4-95f4aa822825"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:47.004672 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004615 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:47.004672 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004631 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:47.004672 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.004640 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:47.006538 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.006507 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b7fc85d-deb8-49fd-b4c4-95f4aa822825" (UID: "8b7fc85d-deb8-49fd-b4c4-95f4aa822825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:07:47.006538 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.006508 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-dshm" (OuterVolumeSpecName: "dshm") pod "8b7fc85d-deb8-49fd-b4c4-95f4aa822825" (UID: "8b7fc85d-deb8-49fd-b4c4-95f4aa822825"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:47.006672 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.006588 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-kube-api-access-rc5nv" (OuterVolumeSpecName: "kube-api-access-rc5nv") pod "8b7fc85d-deb8-49fd-b4c4-95f4aa822825" (UID: "8b7fc85d-deb8-49fd-b4c4-95f4aa822825"). InnerVolumeSpecName "kube-api-access-rc5nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:07:47.105265 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.105237 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:47.105265 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.105262 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc5nv\" (UniqueName: \"kubernetes.io/projected/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-kube-api-access-rc5nv\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:47.105410 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.105272 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7fc85d-deb8-49fd-b4c4-95f4aa822825-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:47.505136 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.505100 2575 generic.go:356] "Generic (PLEG): container finished" podID="8b7fc85d-deb8-49fd-b4c4-95f4aa822825" containerID="0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db" exitCode=0 Aug 05 17:07:47.505335 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.505152 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" Aug 05 17:07:47.505335 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.505196 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" event={"ID":"8b7fc85d-deb8-49fd-b4c4-95f4aa822825","Type":"ContainerDied","Data":"0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db"} Aug 05 17:07:47.505335 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.505240 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx" event={"ID":"8b7fc85d-deb8-49fd-b4c4-95f4aa822825","Type":"ContainerDied","Data":"da69bacfb57270c81afddc8f2ef0f072067e848034d3f6d52bfb79f3b9505865"} Aug 05 17:07:47.505335 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.505258 2575 scope.go:117] "RemoveContainer" containerID="0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db" Aug 05 17:07:47.514616 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.514593 2575 scope.go:117] "RemoveContainer" containerID="0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db" Aug 05 17:07:47.514891 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:07:47.514874 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db\": container with ID starting with 0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db not found: ID does not exist" containerID="0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db" Aug 05 17:07:47.514969 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.514897 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db"} err="failed to get container status \"0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db\": rpc error: code = NotFound desc = could not find container \"0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db\": container with ID starting with 0f47cd32f34aaeacc807c29fbf7c707a040df5d4a8360e4d184df9f2045133db not found: ID does not exist" Aug 05 17:07:47.524170 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.524149 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx"] Aug 05 17:07:47.528668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.528649 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-deb398/test-model-deb398-kserve-55bdc9d55f-x69sx"] Aug 05 17:07:47.588174 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.588153 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" podUID="9db64de6-c670-4ebf-9b97-abfc2656639a" containerName="main" containerID="cri-o://fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1" gracePeriod=45 Aug 05 17:07:47.832519 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.832495 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:07:47.911558 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.911500 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db64de6-c670-4ebf-9b97-abfc2656639a-tls-certs\") pod \"9db64de6-c670-4ebf-9b97-abfc2656639a\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " Aug 05 17:07:47.911558 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.911544 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-tmp-dir\") pod \"9db64de6-c670-4ebf-9b97-abfc2656639a\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " Aug 05 17:07:47.911757 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.911568 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-home\") pod \"9db64de6-c670-4ebf-9b97-abfc2656639a\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " Aug 05 17:07:47.911757 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.911602 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m6fw\" (UniqueName: \"kubernetes.io/projected/9db64de6-c670-4ebf-9b97-abfc2656639a-kube-api-access-7m6fw\") pod \"9db64de6-c670-4ebf-9b97-abfc2656639a\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " Aug 05 17:07:47.911757 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.911627 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-model-cache\") pod \"9db64de6-c670-4ebf-9b97-abfc2656639a\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " Aug 05 17:07:47.911757 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.911656 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-dshm\") pod \"9db64de6-c670-4ebf-9b97-abfc2656639a\" (UID: \"9db64de6-c670-4ebf-9b97-abfc2656639a\") " Aug 05 17:07:47.911968 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.911858 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-home" (OuterVolumeSpecName: "home") pod "9db64de6-c670-4ebf-9b97-abfc2656639a" (UID: "9db64de6-c670-4ebf-9b97-abfc2656639a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:47.911968 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.911873 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9db64de6-c670-4ebf-9b97-abfc2656639a" (UID: "9db64de6-c670-4ebf-9b97-abfc2656639a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:47.911968 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.911888 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9db64de6-c670-4ebf-9b97-abfc2656639a" (UID: "9db64de6-c670-4ebf-9b97-abfc2656639a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:47.912122 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.912076 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:47.912122 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.912095 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:47.912122 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.912109 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:47.913851 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.913826 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db64de6-c670-4ebf-9b97-abfc2656639a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9db64de6-c670-4ebf-9b97-abfc2656639a" (UID: "9db64de6-c670-4ebf-9b97-abfc2656639a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:07:47.913972 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.913953 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-dshm" (OuterVolumeSpecName: "dshm") pod "9db64de6-c670-4ebf-9b97-abfc2656639a" (UID: "9db64de6-c670-4ebf-9b97-abfc2656639a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:47.914033 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:47.913987 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db64de6-c670-4ebf-9b97-abfc2656639a-kube-api-access-7m6fw" (OuterVolumeSpecName: "kube-api-access-7m6fw") pod "9db64de6-c670-4ebf-9b97-abfc2656639a" (UID: "9db64de6-c670-4ebf-9b97-abfc2656639a"). InnerVolumeSpecName "kube-api-access-7m6fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:07:48.013208 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.013176 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db64de6-c670-4ebf-9b97-abfc2656639a-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:48.013336 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.013214 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m6fw\" (UniqueName: \"kubernetes.io/projected/9db64de6-c670-4ebf-9b97-abfc2656639a-kube-api-access-7m6fw\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:48.013336 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.013230 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9db64de6-c670-4ebf-9b97-abfc2656639a-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:48.511571 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.511533 2575 generic.go:356] "Generic (PLEG): container finished" podID="9db64de6-c670-4ebf-9b97-abfc2656639a" containerID="fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1" exitCode=0 Aug 05 17:07:48.511770 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.511577 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" Aug 05 17:07:48.511770 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.511621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" event={"ID":"9db64de6-c670-4ebf-9b97-abfc2656639a","Type":"ContainerDied","Data":"fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1"} Aug 05 17:07:48.511770 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.511660 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8" event={"ID":"9db64de6-c670-4ebf-9b97-abfc2656639a","Type":"ContainerDied","Data":"d3e33f1ef03b31ee0ac93217245051c1b89b043748dfd280c33e84795a2db271"} Aug 05 17:07:48.511770 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.511678 2575 scope.go:117] "RemoveContainer" containerID="fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1" Aug 05 17:07:48.520389 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.520370 2575 scope.go:117] "RemoveContainer" containerID="fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1" Aug 05 17:07:48.520643 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:07:48.520624 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1\": container with ID starting with fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1 not found: ID does not exist" containerID="fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1" Aug 05 17:07:48.520706 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.520648 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1"} err="failed to get container status \"fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1\": rpc error: code = NotFound desc = could not find container \"fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1\": container with ID starting with fec8436195e0158044e71ba9ce11f4fd377e7f44f0367da6b79588ef77aad6a1 not found: ID does not exist" Aug 05 17:07:48.533779 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.533756 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8"] Aug 05 17:07:48.541666 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:48.541648 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3602fc/test-model-3602fc-kserve-896db9689-69bw8"] Aug 05 17:07:49.120992 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:49.120962 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7fc85d-deb8-49fd-b4c4-95f4aa822825" path="/var/lib/kubelet/pods/8b7fc85d-deb8-49fd-b4c4-95f4aa822825/volumes" Aug 05 17:07:49.121423 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:49.121331 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9db64de6-c670-4ebf-9b97-abfc2656639a" path="/var/lib/kubelet/pods/9db64de6-c670-4ebf-9b97-abfc2656639a/volumes" Aug 05 17:07:49.753078 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:49.753039 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:49.859205 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:49.859152 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:50.060693 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.060613 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:50.301709 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.301680 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:50.438027 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.437957 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs"] Aug 05 17:07:50.438335 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.438321 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9db64de6-c670-4ebf-9b97-abfc2656639a" containerName="main" Aug 05 17:07:50.438376 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.438336 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9db64de6-c670-4ebf-9b97-abfc2656639a" containerName="main" Aug 05 17:07:50.438376 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.438354 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7fc85d-deb8-49fd-b4c4-95f4aa822825" containerName="main" Aug 05 17:07:50.438376 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.438359 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7fc85d-deb8-49fd-b4c4-95f4aa822825" containerName="main" Aug 05 17:07:50.438469 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.438430 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9db64de6-c670-4ebf-9b97-abfc2656639a" containerName="main" Aug 05 17:07:50.438469 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.438440 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7fc85d-deb8-49fd-b4c4-95f4aa822825" containerName="main" Aug 05 17:07:50.441287 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.441269 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs" Aug 05 17:07:50.444326 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.444306 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 17:07:50.444436 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.444345 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68wxh\"" Aug 05 17:07:50.456073 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.455434 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs"] Aug 05 17:07:50.532266 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.532233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/466fbc7a-c495-46a4-8d97-cff0071b63a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3602fc-458bs\" (UID: \"466fbc7a-c495-46a4-8d97-cff0071b63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs" Aug 05 17:07:50.633110 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.633082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/466fbc7a-c495-46a4-8d97-cff0071b63a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3602fc-458bs\" (UID: \"466fbc7a-c495-46a4-8d97-cff0071b63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs" Aug 05 17:07:50.633676 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.633655 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/466fbc7a-c495-46a4-8d97-cff0071b63a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3602fc-458bs\" (UID: \"466fbc7a-c495-46a4-8d97-cff0071b63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs" Aug 05 17:07:50.753249 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.753230 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs" Aug 05 17:07:50.889029 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.889001 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs"] Aug 05 17:07:50.891452 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:07:50.891428 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466fbc7a_c495_46a4_8d97_cff0071b63a3.slice/crio-3c6332b100915b3b464f8530ea8d3aab76394380ca09d5829639ac1b1fe95e9a WatchSource:0}: Error finding container 3c6332b100915b3b464f8530ea8d3aab76394380ca09d5829639ac1b1fe95e9a: Status 404 returned error can't find the container with id 3c6332b100915b3b464f8530ea8d3aab76394380ca09d5829639ac1b1fe95e9a Aug 05 17:07:50.893433 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:50.893412 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:07:51.527434 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:51.527400 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs" event={"ID":"466fbc7a-c495-46a4-8d97-cff0071b63a3","Type":"ContainerStarted","Data":"3c6332b100915b3b464f8530ea8d3aab76394380ca09d5829639ac1b1fe95e9a"} Aug 05 17:07:54.544669 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:54.544631 2575 generic.go:356] "Generic (PLEG): container finished" podID="466fbc7a-c495-46a4-8d97-cff0071b63a3" containerID="34f2bd23b6eb3908c4200591d4eed7cfd0d8c7706c69e44fc4d1e85ed3c88baf" exitCode=0 Aug 05 17:07:54.545078 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:54.544704 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs" event={"ID":"466fbc7a-c495-46a4-8d97-cff0071b63a3","Type":"ContainerDied","Data":"34f2bd23b6eb3908c4200591d4eed7cfd0d8c7706c69e44fc4d1e85ed3c88baf"} Aug 05 17:07:55.683118 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:55.683096 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs" Aug 05 17:07:55.780844 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:55.780783 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/466fbc7a-c495-46a4-8d97-cff0071b63a3-maas-api-service-ca\") pod \"466fbc7a-c495-46a4-8d97-cff0071b63a3\" (UID: \"466fbc7a-c495-46a4-8d97-cff0071b63a3\") " Aug 05 17:07:55.781125 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:55.781099 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466fbc7a-c495-46a4-8d97-cff0071b63a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "466fbc7a-c495-46a4-8d97-cff0071b63a3" (UID: "466fbc7a-c495-46a4-8d97-cff0071b63a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:07:55.881654 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:55.881599 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/466fbc7a-c495-46a4-8d97-cff0071b63a3-maas-api-service-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:07:56.555006 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:56.554982 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs" Aug 05 17:07:56.555173 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:56.554982 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs" event={"ID":"466fbc7a-c495-46a4-8d97-cff0071b63a3","Type":"ContainerDied","Data":"3c6332b100915b3b464f8530ea8d3aab76394380ca09d5829639ac1b1fe95e9a"} Aug 05 17:07:56.555173 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:56.555102 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c6332b100915b3b464f8530ea8d3aab76394380ca09d5829639ac1b1fe95e9a" Aug 05 17:07:57.258171 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:57.258146 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 17:07:57.262861 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:07:57.262842 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 17:08:00.943220 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:00.943186 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:01.019208 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:01.019180 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:01.092423 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:01.092391 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:01.133625 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:01.133597 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs"] Aug 05 17:08:01.138600 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:01.138575 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3602fc-458bs"] Aug 05 17:08:01.192550 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:01.192515 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:01.540007 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:01.539952 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:03.046052 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:03.046021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:08:03.120437 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:03.120410 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="466fbc7a-c495-46a4-8d97-cff0071b63a3" path="/var/lib/kubelet/pods/466fbc7a-c495-46a4-8d97-cff0071b63a3/volumes" Aug 05 17:08:07.367992 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:07.367960 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:07.437115 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:07.437080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:07.651745 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:07.651624 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:07.908171 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:07.908078 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:08.291660 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:08.291626 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:08.599852 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:08.599746 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:08.958609 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:08.958521 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:11.345528 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:11.345496 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:11.439015 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:11.438985 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:12.473487 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:12.473450 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:12.579674 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:12.579623 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:12.992085 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:12.992054 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:13.089173 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:13.089144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:14.959309 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:14.959270 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:08:36.331017 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:36.330984 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:36.835764 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:36.835725 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:37.119151 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:37.119034 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:37.590631 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:37.590593 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:40.522783 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:40.522745 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:40.620225 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:40.620191 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:41.663251 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:41.663218 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:42.009130 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:42.009089 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:42.171192 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:42.171161 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:42.271613 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:42.271541 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:44.249460 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:08:44.249430 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:09:05.602014 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:05.601963 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:08.744421 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.744386 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs"] Aug 05 17:09:08.744839 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.744719 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="466fbc7a-c495-46a4-8d97-cff0071b63a3" containerName="revoke-keys" Aug 05 17:09:08.744839 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.744729 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="466fbc7a-c495-46a4-8d97-cff0071b63a3" containerName="revoke-keys" Aug 05 17:09:08.744839 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.744816 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="466fbc7a-c495-46a4-8d97-cff0071b63a3" containerName="revoke-keys" Aug 05 17:09:08.747500 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.747483 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs" Aug 05 17:09:08.750726 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.750705 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 17:09:08.750874 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.750817 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68wxh\"" Aug 05 17:09:08.762810 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.762775 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs"] Aug 05 17:09:08.790193 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.790160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628e842b-3db2-4937-af1b-6a2ec0eaa890-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs\" (UID: \"628e842b-3db2-4937-af1b-6a2ec0eaa890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs" Aug 05 17:09:08.891046 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.891007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628e842b-3db2-4937-af1b-6a2ec0eaa890-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs\" (UID: \"628e842b-3db2-4937-af1b-6a2ec0eaa890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs" Aug 05 17:09:08.891592 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:08.891571 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628e842b-3db2-4937-af1b-6a2ec0eaa890-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs\" (UID: \"628e842b-3db2-4937-af1b-6a2ec0eaa890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs" Aug 05 17:09:09.057011 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:09.056930 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs" Aug 05 17:09:09.190306 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:09.190285 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs"] Aug 05 17:09:09.192148 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:09:09.192116 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628e842b_3db2_4937_af1b_6a2ec0eaa890.slice/crio-4ff8972c7676943040543291d66915291ecaec16b826b02ed22ec0a82ec05801 WatchSource:0}: Error finding container 4ff8972c7676943040543291d66915291ecaec16b826b02ed22ec0a82ec05801: Status 404 returned error can't find the container with id 4ff8972c7676943040543291d66915291ecaec16b826b02ed22ec0a82ec05801 Aug 05 17:09:09.906149 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:09.906124 2575 generic.go:356] "Generic (PLEG): container finished" podID="628e842b-3db2-4937-af1b-6a2ec0eaa890" containerID="ee599cf76637c6893d5a6ccbab34445637a7864317a8a184e8bd6c9c027d8151" exitCode=0 Aug 05 17:09:09.906427 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:09.906206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs" event={"ID":"628e842b-3db2-4937-af1b-6a2ec0eaa890","Type":"ContainerDied","Data":"ee599cf76637c6893d5a6ccbab34445637a7864317a8a184e8bd6c9c027d8151"} Aug 05 17:09:09.906427 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:09.906260 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs" event={"ID":"628e842b-3db2-4937-af1b-6a2ec0eaa890","Type":"ContainerStarted","Data":"4ff8972c7676943040543291d66915291ecaec16b826b02ed22ec0a82ec05801"} Aug 05 17:09:11.038781 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:11.038756 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs" Aug 05 17:09:11.106245 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:11.106222 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628e842b-3db2-4937-af1b-6a2ec0eaa890-maas-api-service-ca\") pod \"628e842b-3db2-4937-af1b-6a2ec0eaa890\" (UID: \"628e842b-3db2-4937-af1b-6a2ec0eaa890\") " Aug 05 17:09:11.106531 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:11.106511 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628e842b-3db2-4937-af1b-6a2ec0eaa890-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "628e842b-3db2-4937-af1b-6a2ec0eaa890" (UID: "628e842b-3db2-4937-af1b-6a2ec0eaa890"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:09:11.207939 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:11.207879 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628e842b-3db2-4937-af1b-6a2ec0eaa890-maas-api-service-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:09:11.916928 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:11.916894 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs" event={"ID":"628e842b-3db2-4937-af1b-6a2ec0eaa890","Type":"ContainerDied","Data":"4ff8972c7676943040543291d66915291ecaec16b826b02ed22ec0a82ec05801"} Aug 05 17:09:11.916928 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:11.916916 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs" Aug 05 17:09:11.917125 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:11.916942 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ff8972c7676943040543291d66915291ecaec16b826b02ed22ec0a82ec05801" Aug 05 17:09:24.415548 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:24.415441 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:24.506853 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:24.506823 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:24.581091 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:24.581062 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:24.664122 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:24.664090 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs"] Aug 05 17:09:24.667243 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:24.667168 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e153a-fsgqs"] Aug 05 17:09:24.668876 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:24.668804 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:25.120352 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:25.120319 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="628e842b-3db2-4937-af1b-6a2ec0eaa890" path="/var/lib/kubelet/pods/628e842b-3db2-4937-af1b-6a2ec0eaa890/volumes" Aug 05 17:09:25.147063 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:25.147036 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:27.176480 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:27.176449 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:09:31.137692 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.137643 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:31.296972 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.296936 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:31.498300 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.498262 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:31.740670 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.740633 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:31.870455 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.870390 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz"] Aug 05 17:09:31.870749 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.870737 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628e842b-3db2-4937-af1b-6a2ec0eaa890" containerName="revoke-keys" Aug 05 17:09:31.870810 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.870751 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="628e842b-3db2-4937-af1b-6a2ec0eaa890" containerName="revoke-keys" Aug 05 17:09:31.870852 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.870818 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="628e842b-3db2-4937-af1b-6a2ec0eaa890" containerName="revoke-keys" Aug 05 17:09:31.873653 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.873637 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz" Aug 05 17:09:31.876428 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.876409 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 17:09:31.876589 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.876574 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68wxh\"" Aug 05 17:09:31.885596 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.885576 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz"] Aug 05 17:09:31.982386 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:31.982356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94f84f2-ffe3-42f4-84c4-8e68073e5d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz\" (UID: \"d94f84f2-ffe3-42f4-84c4-8e68073e5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz" Aug 05 17:09:32.082995 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:32.082969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94f84f2-ffe3-42f4-84c4-8e68073e5d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz\" (UID: \"d94f84f2-ffe3-42f4-84c4-8e68073e5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz" Aug 05 17:09:32.083583 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:32.083566 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94f84f2-ffe3-42f4-84c4-8e68073e5d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz\" (UID: \"d94f84f2-ffe3-42f4-84c4-8e68073e5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz" Aug 05 17:09:32.184179 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:32.184121 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz" Aug 05 17:09:32.313344 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:32.313319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz"] Aug 05 17:09:32.315564 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:09:32.315536 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94f84f2_ffe3_42f4_84c4_8e68073e5d82.slice/crio-7d8512ecdea388eb267892394812551382cd8cbb4c5563e547cbb9473f8e4749 WatchSource:0}: Error finding container 7d8512ecdea388eb267892394812551382cd8cbb4c5563e547cbb9473f8e4749: Status 404 returned error can't find the container with id 7d8512ecdea388eb267892394812551382cd8cbb4c5563e547cbb9473f8e4749 Aug 05 17:09:33.020745 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:33.020716 2575 generic.go:356] "Generic (PLEG): container finished" podID="d94f84f2-ffe3-42f4-84c4-8e68073e5d82" containerID="297ba24605a555ea99afeb623c6cd954ccb420442977cdfd390e05fe13a2be21" exitCode=0 Aug 05 17:09:33.020890 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:33.020812 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz" event={"ID":"d94f84f2-ffe3-42f4-84c4-8e68073e5d82","Type":"ContainerDied","Data":"297ba24605a555ea99afeb623c6cd954ccb420442977cdfd390e05fe13a2be21"} Aug 05 17:09:33.020890 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:33.020885 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz" event={"ID":"d94f84f2-ffe3-42f4-84c4-8e68073e5d82","Type":"ContainerStarted","Data":"7d8512ecdea388eb267892394812551382cd8cbb4c5563e547cbb9473f8e4749"} Aug 05 17:09:34.154119 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:34.154096 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz" Aug 05 17:09:34.301260 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:34.301186 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94f84f2-ffe3-42f4-84c4-8e68073e5d82-maas-api-service-ca\") pod \"d94f84f2-ffe3-42f4-84c4-8e68073e5d82\" (UID: \"d94f84f2-ffe3-42f4-84c4-8e68073e5d82\") " Aug 05 17:09:34.301554 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:34.301529 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94f84f2-ffe3-42f4-84c4-8e68073e5d82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d94f84f2-ffe3-42f4-84c4-8e68073e5d82" (UID: "d94f84f2-ffe3-42f4-84c4-8e68073e5d82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:09:34.402143 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:34.402115 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94f84f2-ffe3-42f4-84c4-8e68073e5d82-maas-api-service-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:09:35.031664 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:35.031636 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz" Aug 05 17:09:35.031664 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:35.031651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz" event={"ID":"d94f84f2-ffe3-42f4-84c4-8e68073e5d82","Type":"ContainerDied","Data":"7d8512ecdea388eb267892394812551382cd8cbb4c5563e547cbb9473f8e4749"} Aug 05 17:09:35.031943 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:35.031720 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d8512ecdea388eb267892394812551382cd8cbb4c5563e547cbb9473f8e4749" Aug 05 17:09:42.800417 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:42.800381 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:42.930622 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:42.930584 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:43.037004 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:43.036962 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:43.147893 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:43.147780 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:43.157034 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:43.157010 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz"] Aug 05 17:09:43.161909 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:43.161888 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eaba9-dm4wz"] Aug 05 17:09:43.571080 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:43.571049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:45.056934 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:45.056903 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:09:45.120505 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:45.120478 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d94f84f2-ffe3-42f4-84c4-8e68073e5d82" path="/var/lib/kubelet/pods/d94f84f2-ffe3-42f4-84c4-8e68073e5d82/volumes" Aug 05 17:09:49.772148 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:49.772111 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:49.872180 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:49.872144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:50.066576 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:50.066470 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:50.334599 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:50.334517 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:50.702096 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:50.702017 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:51.073975 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:51.073942 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:51.853545 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:51.853514 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:54.182574 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:54.182536 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:54.312701 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:54.312669 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:55.413753 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:55.413722 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:55.605257 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:55.605214 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:56.162174 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:56.162144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:56.256658 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:56.256632 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:58.386929 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:58.381742 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:09:59.930616 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:09:59.930546 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:19.188230 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:19.188196 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:19.466776 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:19.466702 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:19.874702 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:19.874664 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:20.075013 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:20.074980 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:23.182954 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:23.182924 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:23.284471 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:23.284442 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:24.352420 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:24.352389 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:24.771228 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:24.771196 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:24.929997 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:24.929959 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:25.030747 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:25.030659 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:26.953130 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:26.953093 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:10:46.514908 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:46.514873 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:46.612128 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:46.612080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:47.961369 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:47.961299 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:48.259199 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:10:48.259167 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:11:07.249149 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.249120 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r"] Aug 05 17:11:07.249543 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.249460 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d94f84f2-ffe3-42f4-84c4-8e68073e5d82" containerName="revoke-keys" Aug 05 17:11:07.249543 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.249470 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d94f84f2-ffe3-42f4-84c4-8e68073e5d82" containerName="revoke-keys" Aug 05 17:11:07.249614 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.249572 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d94f84f2-ffe3-42f4-84c4-8e68073e5d82" containerName="revoke-keys" Aug 05 17:11:07.252552 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.252536 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.255497 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.255477 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-92434d\"/\"kube-root-ca.crt\"" Aug 05 17:11:07.256646 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.256630 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-92434d\"/\"openshift-service-ca.crt\"" Aug 05 17:11:07.256786 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.256724 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-92434d\"/\"default-dockercfg-qm4gt\"" Aug 05 17:11:07.256786 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.256731 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-92434d\"/\"auth-test-model-92434d-kserve-self-signed-certs\"" Aug 05 17:11:07.262988 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.262967 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r"] Aug 05 17:11:07.327478 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.327452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-home\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.327576 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.327488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-267jf\" (UniqueName: \"kubernetes.io/projected/bd673006-f0b5-462d-aa80-5485601d0bd1-kube-api-access-267jf\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.327576 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.327507 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-tmp-dir\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.327648 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.327607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-dshm\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.327648 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.327639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-model-cache\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.327711 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.327676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd673006-f0b5-462d-aa80-5485601d0bd1-tls-certs\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.336646 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.336629 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:11:07.428067 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.428041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-dshm\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.428170 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.428072 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-model-cache\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.428170 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.428096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd673006-f0b5-462d-aa80-5485601d0bd1-tls-certs\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.428170 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.428118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-home\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.428170 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.428141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-267jf\" (UniqueName: \"kubernetes.io/projected/bd673006-f0b5-462d-aa80-5485601d0bd1-kube-api-access-267jf\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.428170 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.428158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-tmp-dir\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.428516 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.428495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-model-cache\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.428631 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.428532 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-home\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.428631 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.428570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-tmp-dir\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.430400 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.430380 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-dshm\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.430569 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.430550 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd673006-f0b5-462d-aa80-5485601d0bd1-tls-certs\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.440273 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.440244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-267jf\" (UniqueName: \"kubernetes.io/projected/bd673006-f0b5-462d-aa80-5485601d0bd1-kube-api-access-267jf\") pod \"auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.561454 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.561363 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:07.703390 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:07.703364 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r"] Aug 05 17:11:07.704928 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:11:07.704901 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd673006_f0b5_462d_aa80_5485601d0bd1.slice/crio-485f2932fe94cc172b147bcc3b766ed2ed9b69d2b790ef0a3f5f57d767068edc WatchSource:0}: Error finding container 485f2932fe94cc172b147bcc3b766ed2ed9b69d2b790ef0a3f5f57d767068edc: Status 404 returned error can't find the container with id 485f2932fe94cc172b147bcc3b766ed2ed9b69d2b790ef0a3f5f57d767068edc Aug 05 17:11:08.478739 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:08.478698 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" event={"ID":"bd673006-f0b5-462d-aa80-5485601d0bd1","Type":"ContainerStarted","Data":"293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940"} Aug 05 17:11:08.478739 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:08.478744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" event={"ID":"bd673006-f0b5-462d-aa80-5485601d0bd1","Type":"ContainerStarted","Data":"485f2932fe94cc172b147bcc3b766ed2ed9b69d2b790ef0a3f5f57d767068edc"} Aug 05 17:11:08.499363 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:08.499315 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" podStartSLOduration=1.499302103 podStartE2EDuration="1.499302103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:11:08.499215134 +0000 UTC m=+2891.858466979" watchObservedRunningTime="2026-08-05 17:11:08.499302103 +0000 UTC m=+2891.858553925" Aug 05 17:11:09.960961 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:09.960929 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:11:17.562601 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:17.562572 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:17.563129 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:17.562620 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:17.575004 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:17.574975 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:18.539113 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:18.539085 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:11:28.963364 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:28.963330 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f"] Aug 05 17:11:28.967001 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:28.966977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:28.970017 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:28.969999 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"auth-test-model-57997e-kserve-self-signed-certs\"" Aug 05 17:11:28.971423 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:28.971402 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"default-dockercfg-klcgc\"" Aug 05 17:11:28.971522 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:28.971445 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"kube-root-ca.crt\"" Aug 05 17:11:28.971522 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:28.971409 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"openshift-service-ca.crt\"" Aug 05 17:11:28.984390 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:28.984365 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f"] Aug 05 17:11:29.086290 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.086257 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:11:29.099450 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.099427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-model-cache\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.099529 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.099456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-tmp-dir\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.099529 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.099475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7rt\" (UniqueName: \"kubernetes.io/projected/f7afa752-6b47-4db8-a640-4206617ce9f5-kube-api-access-8h7rt\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.099529 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.099511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-home\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.099636 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.099552 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7afa752-6b47-4db8-a640-4206617ce9f5-tls-certs\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.099636 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.099594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-dshm\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.201090 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.201038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-model-cache\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.201090 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.201099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-tmp-dir\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.201364 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.201126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7rt\" (UniqueName: \"kubernetes.io/projected/f7afa752-6b47-4db8-a640-4206617ce9f5-kube-api-access-8h7rt\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.201364 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.201162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-home\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.201364 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.201201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7afa752-6b47-4db8-a640-4206617ce9f5-tls-certs\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.201364 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.201245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-dshm\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.201585 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.201502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-model-cache\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.201643 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.201597 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-tmp-dir\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.201643 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.201623 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-home\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.203912 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.203882 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-dshm\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.204057 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.203985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7afa752-6b47-4db8-a640-4206617ce9f5-tls-certs\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.211094 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.211066 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7rt\" (UniqueName: \"kubernetes.io/projected/f7afa752-6b47-4db8-a640-4206617ce9f5-kube-api-access-8h7rt\") pod \"auth-test-model-57997e-kserve-785694c76c-crs5f\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.276498 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.276455 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:29.411221 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.411195 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f"] Aug 05 17:11:29.413306 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:11:29.413270 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7afa752_6b47_4db8_a640_4206617ce9f5.slice/crio-a943a6bf96a9b9e2052f7cc09f5293d73f42a8f0b058cc867220e92ce937f105 WatchSource:0}: Error finding container a943a6bf96a9b9e2052f7cc09f5293d73f42a8f0b058cc867220e92ce937f105: Status 404 returned error can't find the container with id a943a6bf96a9b9e2052f7cc09f5293d73f42a8f0b058cc867220e92ce937f105 Aug 05 17:11:29.581924 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.581826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" event={"ID":"f7afa752-6b47-4db8-a640-4206617ce9f5","Type":"ContainerStarted","Data":"1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586"} Aug 05 17:11:29.581924 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.581872 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" event={"ID":"f7afa752-6b47-4db8-a640-4206617ce9f5","Type":"ContainerStarted","Data":"a943a6bf96a9b9e2052f7cc09f5293d73f42a8f0b058cc867220e92ce937f105"} Aug 05 17:11:29.601234 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:29.601188 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" podStartSLOduration=1.60116011 podStartE2EDuration="1.60116011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:11:29.600145732 +0000 UTC m=+2912.959397578" watchObservedRunningTime="2026-08-05 17:11:29.60116011 +0000 UTC m=+2912.960411997" Aug 05 17:11:31.952266 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:31.952233 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:11:39.276996 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:39.276948 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:39.276996 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:39.277010 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:39.289483 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:39.289461 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:39.643518 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:39.643439 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:11:53.956940 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:11:53.956905 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:12:09.256591 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:12:09.256553 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:12:20.551036 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:12:20.550947 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:12:35.861922 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:12:35.861886 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:12:44.215961 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:12:44.215922 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:12:51.554091 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:12:51.554059 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:12:57.290033 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:12:57.289993 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 17:12:57.295132 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:12:57.295113 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 17:13:02.149413 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:13:02.149372 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:13:05.998133 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:13:05.998096 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:13:17.847674 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:13:17.847637 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:13:41.752947 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:13:41.752913 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:13:53.054065 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:13:53.053961 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:13:57.363672 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:13:57.363639 2575 scope.go:117] "RemoveContainer" containerID="34f2bd23b6eb3908c4200591d4eed7cfd0d8c7706c69e44fc4d1e85ed3c88baf" Aug 05 17:14:08.056549 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:14:08.056518 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:14:23.753922 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:14:23.753883 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:14:34.553114 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:14:34.553072 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:14:50.150455 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:14:50.150416 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:15:07.443354 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:07.443321 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:15:17.542704 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:17.542670 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:15:32.448829 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:32.448776 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:15:36.424684 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:36.424646 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r"] Aug 05 17:15:36.472557 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:36.472511 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:15:37.362050 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:37.362013 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f"] Aug 05 17:15:37.381154 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:37.380398 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:15:43.253034 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:43.252999 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:15:46.470489 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.470450 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p"] Aug 05 17:15:46.473903 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.473885 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.476721 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.476688 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-92434d\"/\"sub-test-model-92434d-kserve-self-signed-certs\"" Aug 05 17:15:46.484250 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.484224 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p"] Aug 05 17:15:46.541940 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.541902 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:15:46.550044 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.550015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdbe929-2a37-4231-abbe-b086cfd6677b-tls-certs\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.550164 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.550054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-tmp-dir\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.550164 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.550138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-home\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.550240 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.550175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-dshm\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.550240 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.550216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-model-cache\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.550304 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.550245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctbz7\" (UniqueName: \"kubernetes.io/projected/dcdbe929-2a37-4231-abbe-b086cfd6677b-kube-api-access-ctbz7\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.651455 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.651417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-model-cache\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.651455 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.651460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctbz7\" (UniqueName: \"kubernetes.io/projected/dcdbe929-2a37-4231-abbe-b086cfd6677b-kube-api-access-ctbz7\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.651701 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.651510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdbe929-2a37-4231-abbe-b086cfd6677b-tls-certs\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.651701 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.651530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-tmp-dir\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.651701 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.651564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-home\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.651701 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.651638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-dshm\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.651963 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.651896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-model-cache\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.651963 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.651936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-home\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.652071 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.652006 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-tmp-dir\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.654070 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.654048 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-dshm\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.654312 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.654292 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdbe929-2a37-4231-abbe-b086cfd6677b-tls-certs\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.661101 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.661072 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctbz7\" (UniqueName: \"kubernetes.io/projected/dcdbe929-2a37-4231-abbe-b086cfd6677b-kube-api-access-ctbz7\") pod \"sub-test-model-92434d-kserve-768cbd75bc-lbx8p\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.783884 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.783836 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:46.924972 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:46.924940 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p"] Aug 05 17:15:46.926874 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:15:46.926842 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcdbe929_2a37_4231_abbe_b086cfd6677b.slice/crio-6a899b6e17657fbe6a48502ed3cd461aa2382a65233d2413570c94e1d9b4f1be WatchSource:0}: Error finding container 6a899b6e17657fbe6a48502ed3cd461aa2382a65233d2413570c94e1d9b4f1be: Status 404 returned error can't find the container with id 6a899b6e17657fbe6a48502ed3cd461aa2382a65233d2413570c94e1d9b4f1be Aug 05 17:15:47.862440 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:47.862404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" event={"ID":"dcdbe929-2a37-4231-abbe-b086cfd6677b","Type":"ContainerStarted","Data":"776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2"} Aug 05 17:15:47.862841 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:47.862449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" event={"ID":"dcdbe929-2a37-4231-abbe-b086cfd6677b","Type":"ContainerStarted","Data":"6a899b6e17657fbe6a48502ed3cd461aa2382a65233d2413570c94e1d9b4f1be"} Aug 05 17:15:47.882482 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:47.882435 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" podStartSLOduration=1.882417222 podStartE2EDuration="1.882417222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:47.880522916 +0000 UTC m=+3171.239774760" watchObservedRunningTime="2026-08-05 17:15:47.882417222 +0000 UTC m=+3171.241669066" Aug 05 17:15:49.242370 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:49.242332 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:15:51.445360 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.445285 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" podUID="bd673006-f0b5-462d-aa80-5485601d0bd1" containerName="main" containerID="cri-o://293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940" gracePeriod=45 Aug 05 17:15:51.697260 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.697186 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:15:51.798368 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798316 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-tmp-dir\") pod \"bd673006-f0b5-462d-aa80-5485601d0bd1\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " Aug 05 17:15:51.798568 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798386 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-home\") pod \"bd673006-f0b5-462d-aa80-5485601d0bd1\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " Aug 05 17:15:51.798568 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798421 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-267jf\" (UniqueName: \"kubernetes.io/projected/bd673006-f0b5-462d-aa80-5485601d0bd1-kube-api-access-267jf\") pod \"bd673006-f0b5-462d-aa80-5485601d0bd1\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " Aug 05 17:15:51.798568 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798446 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd673006-f0b5-462d-aa80-5485601d0bd1-tls-certs\") pod \"bd673006-f0b5-462d-aa80-5485601d0bd1\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " Aug 05 17:15:51.798568 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798470 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-dshm\") pod \"bd673006-f0b5-462d-aa80-5485601d0bd1\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " Aug 05 17:15:51.798568 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798506 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-model-cache\") pod \"bd673006-f0b5-462d-aa80-5485601d0bd1\" (UID: \"bd673006-f0b5-462d-aa80-5485601d0bd1\") " Aug 05 17:15:51.798860 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798594 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd673006-f0b5-462d-aa80-5485601d0bd1" (UID: "bd673006-f0b5-462d-aa80-5485601d0bd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:51.798860 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798676 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-home" (OuterVolumeSpecName: "home") pod "bd673006-f0b5-462d-aa80-5485601d0bd1" (UID: "bd673006-f0b5-462d-aa80-5485601d0bd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:51.798860 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798820 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd673006-f0b5-462d-aa80-5485601d0bd1" (UID: "bd673006-f0b5-462d-aa80-5485601d0bd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:51.798977 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798906 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:51.798977 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.798927 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:51.800892 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.800840 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-dshm" (OuterVolumeSpecName: "dshm") pod "bd673006-f0b5-462d-aa80-5485601d0bd1" (UID: "bd673006-f0b5-462d-aa80-5485601d0bd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:51.800892 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.800847 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd673006-f0b5-462d-aa80-5485601d0bd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd673006-f0b5-462d-aa80-5485601d0bd1" (UID: "bd673006-f0b5-462d-aa80-5485601d0bd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:15:51.801045 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.800946 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd673006-f0b5-462d-aa80-5485601d0bd1-kube-api-access-267jf" (OuterVolumeSpecName: "kube-api-access-267jf") pod "bd673006-f0b5-462d-aa80-5485601d0bd1" (UID: "bd673006-f0b5-462d-aa80-5485601d0bd1"). InnerVolumeSpecName "kube-api-access-267jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:15:51.883515 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.883483 2575 generic.go:356] "Generic (PLEG): container finished" podID="bd673006-f0b5-462d-aa80-5485601d0bd1" containerID="293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940" exitCode=0 Aug 05 17:15:51.883687 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.883533 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" Aug 05 17:15:51.883687 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.883573 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" event={"ID":"bd673006-f0b5-462d-aa80-5485601d0bd1","Type":"ContainerDied","Data":"293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940"} Aug 05 17:15:51.883687 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.883619 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r" event={"ID":"bd673006-f0b5-462d-aa80-5485601d0bd1","Type":"ContainerDied","Data":"485f2932fe94cc172b147bcc3b766ed2ed9b69d2b790ef0a3f5f57d767068edc"} Aug 05 17:15:51.883687 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.883640 2575 scope.go:117] "RemoveContainer" containerID="293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940" Aug 05 17:15:51.893180 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.893155 2575 scope.go:117] "RemoveContainer" containerID="293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940" Aug 05 17:15:51.893459 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:15:51.893440 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940\": container with ID starting with 293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940 not found: ID does not exist" containerID="293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940" Aug 05 17:15:51.893517 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.893468 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940"} err="failed to get container status \"293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940\": rpc error: code = NotFound desc = could not find container \"293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940\": container with ID starting with 293ecf2d1e57a213566b7d09d9794e020980a139f99a58b64a6716bbc033d940 not found: ID does not exist" Aug 05 17:15:51.899854 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.899829 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:51.899854 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.899854 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd673006-f0b5-462d-aa80-5485601d0bd1-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:51.900007 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.899869 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-267jf\" (UniqueName: \"kubernetes.io/projected/bd673006-f0b5-462d-aa80-5485601d0bd1-kube-api-access-267jf\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:51.900007 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.899881 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd673006-f0b5-462d-aa80-5485601d0bd1-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:51.907665 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.907638 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r"] Aug 05 17:15:51.913143 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:51.913119 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/auth-test-model-92434d-kserve-dcfd5d7d7-nxp6r"] Aug 05 17:15:52.386137 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.386077 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" podUID="f7afa752-6b47-4db8-a640-4206617ce9f5" containerName="main" containerID="cri-o://1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586" gracePeriod=45 Aug 05 17:15:52.638368 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.638300 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:15:52.707347 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707302 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-tmp-dir\") pod \"f7afa752-6b47-4db8-a640-4206617ce9f5\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " Aug 05 17:15:52.707521 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707364 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-dshm\") pod \"f7afa752-6b47-4db8-a640-4206617ce9f5\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " Aug 05 17:15:52.707521 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707426 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-model-cache\") pod \"f7afa752-6b47-4db8-a640-4206617ce9f5\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " Aug 05 17:15:52.707521 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707462 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h7rt\" (UniqueName: \"kubernetes.io/projected/f7afa752-6b47-4db8-a640-4206617ce9f5-kube-api-access-8h7rt\") pod \"f7afa752-6b47-4db8-a640-4206617ce9f5\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " Aug 05 17:15:52.707521 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707509 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7afa752-6b47-4db8-a640-4206617ce9f5-tls-certs\") pod \"f7afa752-6b47-4db8-a640-4206617ce9f5\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " Aug 05 17:15:52.707734 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707549 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-home\") pod \"f7afa752-6b47-4db8-a640-4206617ce9f5\" (UID: \"f7afa752-6b47-4db8-a640-4206617ce9f5\") " Aug 05 17:15:52.707734 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707609 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7afa752-6b47-4db8-a640-4206617ce9f5" (UID: "f7afa752-6b47-4db8-a640-4206617ce9f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:52.707869 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707763 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7afa752-6b47-4db8-a640-4206617ce9f5" (UID: "f7afa752-6b47-4db8-a640-4206617ce9f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:52.707945 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707864 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-home" (OuterVolumeSpecName: "home") pod "f7afa752-6b47-4db8-a640-4206617ce9f5" (UID: "f7afa752-6b47-4db8-a640-4206617ce9f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:52.707945 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707893 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:52.707945 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.707909 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:52.709889 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.709859 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7afa752-6b47-4db8-a640-4206617ce9f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7afa752-6b47-4db8-a640-4206617ce9f5" (UID: "f7afa752-6b47-4db8-a640-4206617ce9f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:15:52.709889 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.709894 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-dshm" (OuterVolumeSpecName: "dshm") pod "f7afa752-6b47-4db8-a640-4206617ce9f5" (UID: "f7afa752-6b47-4db8-a640-4206617ce9f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:52.710052 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.710010 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7afa752-6b47-4db8-a640-4206617ce9f5-kube-api-access-8h7rt" (OuterVolumeSpecName: "kube-api-access-8h7rt") pod "f7afa752-6b47-4db8-a640-4206617ce9f5" (UID: "f7afa752-6b47-4db8-a640-4206617ce9f5"). InnerVolumeSpecName "kube-api-access-8h7rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:15:52.809410 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.809362 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h7rt\" (UniqueName: \"kubernetes.io/projected/f7afa752-6b47-4db8-a640-4206617ce9f5-kube-api-access-8h7rt\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:52.809410 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.809400 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7afa752-6b47-4db8-a640-4206617ce9f5-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:52.809410 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.809414 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:52.809650 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.809426 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7afa752-6b47-4db8-a640-4206617ce9f5-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:15:52.891011 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.890917 2575 generic.go:356] "Generic (PLEG): container finished" podID="f7afa752-6b47-4db8-a640-4206617ce9f5" containerID="1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586" exitCode=0 Aug 05 17:15:52.891011 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.890963 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" Aug 05 17:15:52.891011 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.890986 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" event={"ID":"f7afa752-6b47-4db8-a640-4206617ce9f5","Type":"ContainerDied","Data":"1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586"} Aug 05 17:15:52.891274 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.891017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f" event={"ID":"f7afa752-6b47-4db8-a640-4206617ce9f5","Type":"ContainerDied","Data":"a943a6bf96a9b9e2052f7cc09f5293d73f42a8f0b058cc867220e92ce937f105"} Aug 05 17:15:52.891274 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.891032 2575 scope.go:117] "RemoveContainer" containerID="1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586" Aug 05 17:15:52.900555 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.900533 2575 scope.go:117] "RemoveContainer" containerID="1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586" Aug 05 17:15:52.900907 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:15:52.900887 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586\": container with ID starting with 1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586 not found: ID does not exist" containerID="1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586" Aug 05 17:15:52.900978 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.900918 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586"} err="failed to get container status \"1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586\": rpc error: code = NotFound desc = could not find container \"1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586\": container with ID starting with 1b4023f9bc8be42abfe3c8a6f7bac327435aed35788d445c85f8241b6a016586 not found: ID does not exist" Aug 05 17:15:52.912692 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.912661 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f"] Aug 05 17:15:52.916849 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:52.916816 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/auth-test-model-57997e-kserve-785694c76c-crs5f"] Aug 05 17:15:53.119943 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:53.119913 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd673006-f0b5-462d-aa80-5485601d0bd1" path="/var/lib/kubelet/pods/bd673006-f0b5-462d-aa80-5485601d0bd1/volumes" Aug 05 17:15:53.120348 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:53.120334 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7afa752-6b47-4db8-a640-4206617ce9f5" path="/var/lib/kubelet/pods/f7afa752-6b47-4db8-a640-4206617ce9f5/volumes" Aug 05 17:15:56.784011 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:56.783956 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:56.784439 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:56.784054 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:56.796907 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:56.796851 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:56.922364 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:56.922329 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:15:57.391567 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:57.391536 2575 scope.go:117] "RemoveContainer" containerID="297ba24605a555ea99afeb623c6cd954ccb420442977cdfd390e05fe13a2be21" Aug 05 17:15:57.401544 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:15:57.401512 2575 scope.go:117] "RemoveContainer" containerID="ee599cf76637c6893d5a6ccbab34445637a7864317a8a184e8bd6c9c027d8151" Aug 05 17:16:08.248787 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.248753 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f"] Aug 05 17:16:08.249214 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.249150 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd673006-f0b5-462d-aa80-5485601d0bd1" containerName="main" Aug 05 17:16:08.249214 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.249162 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd673006-f0b5-462d-aa80-5485601d0bd1" containerName="main" Aug 05 17:16:08.249214 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.249175 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7afa752-6b47-4db8-a640-4206617ce9f5" containerName="main" Aug 05 17:16:08.249214 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.249180 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7afa752-6b47-4db8-a640-4206617ce9f5" containerName="main" Aug 05 17:16:08.249348 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.249248 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd673006-f0b5-462d-aa80-5485601d0bd1" containerName="main" Aug 05 17:16:08.249348 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.249257 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7afa752-6b47-4db8-a640-4206617ce9f5" containerName="main" Aug 05 17:16:08.252179 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.252161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.255319 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.255296 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"kube-root-ca.crt\"" Aug 05 17:16:08.256907 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.256884 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"openshift-service-ca.crt\"" Aug 05 17:16:08.257143 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.256899 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"default-dockercfg-klcgc\"" Aug 05 17:16:08.257264 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.256898 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"sub-test-model-57997e-kserve-self-signed-certs\"" Aug 05 17:16:08.262128 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.262105 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f"] Aug 05 17:16:08.308223 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.308187 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:16:08.352198 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.352157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-home\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.352407 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.352207 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-dshm\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.352407 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.352285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gh7\" (UniqueName: \"kubernetes.io/projected/f33d449e-a952-425e-bf8a-59651551f2c0-kube-api-access-79gh7\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.352407 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.352328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f33d449e-a952-425e-bf8a-59651551f2c0-tls-certs\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.352544 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.352415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-tmp-dir\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.352544 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.352452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-model-cache\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.452998 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.452960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-model-cache\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.453176 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.453014 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-home\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.453176 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.453046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-dshm\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.453176 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.453081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gh7\" (UniqueName: \"kubernetes.io/projected/f33d449e-a952-425e-bf8a-59651551f2c0-kube-api-access-79gh7\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.453176 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.453107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f33d449e-a952-425e-bf8a-59651551f2c0-tls-certs\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.453176 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.453148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-tmp-dir\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.453481 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.453443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-model-cache\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.453564 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.453484 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-home\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.453564 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.453549 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-tmp-dir\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.455777 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.455738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-dshm\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.455955 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.455920 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f33d449e-a952-425e-bf8a-59651551f2c0-tls-certs\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.462400 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.462372 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gh7\" (UniqueName: \"kubernetes.io/projected/f33d449e-a952-425e-bf8a-59651551f2c0-kube-api-access-79gh7\") pod \"sub-test-model-57997e-kserve-967d5b8dc-hnx4f\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.562973 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.562870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:08.703189 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.703162 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f"] Aug 05 17:16:08.705386 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:16:08.705351 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf33d449e_a952_425e_bf8a_59651551f2c0.slice/crio-edee76b6f645bae11e4e575cd42d12087749e8c37cf02ce9601a3e430612d957 WatchSource:0}: Error finding container edee76b6f645bae11e4e575cd42d12087749e8c37cf02ce9601a3e430612d957: Status 404 returned error can't find the container with id edee76b6f645bae11e4e575cd42d12087749e8c37cf02ce9601a3e430612d957 Aug 05 17:16:08.979069 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.978974 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" event={"ID":"f33d449e-a952-425e-bf8a-59651551f2c0","Type":"ContainerStarted","Data":"cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748"} Aug 05 17:16:08.979069 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.979021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" event={"ID":"f33d449e-a952-425e-bf8a-59651551f2c0","Type":"ContainerStarted","Data":"edee76b6f645bae11e4e575cd42d12087749e8c37cf02ce9601a3e430612d957"} Aug 05 17:16:08.996713 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:08.996622 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" podStartSLOduration=0.996603225 podStartE2EDuration="996.603225ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:16:08.995952636 +0000 UTC m=+3192.355204493" watchObservedRunningTime="2026-08-05 17:16:08.996603225 +0000 UTC m=+3192.355855068" Aug 05 17:16:09.248943 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:09.248904 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:16:10.752007 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:10.751969 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:16:18.563031 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:18.562982 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:18.563590 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:18.563046 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:18.575882 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:18.575843 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:19.039490 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:19.039453 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:16:40.857566 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:40.857528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:16:40.958345 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:40.958300 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:16:44.750368 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:44.750332 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:16:55.706055 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:55.705956 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:16:55.804945 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:55.804909 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:16:57.805158 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:16:57.805114 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:17:05.878547 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:05.878509 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:06.015294 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:06.015249 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:09.960281 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:09.960245 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:17:15.565292 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:15.565253 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p"] Aug 05 17:17:15.593660 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:15.593613 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:16.536453 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:16.536417 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f"] Aug 05 17:17:16.559967 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:16.559932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:19.557978 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:19.557938 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:17:24.060165 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.060117 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r"] Aug 05 17:17:24.063611 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.063588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.067318 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.067295 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-92434d\"/\"rate-test-model-92434d-kserve-self-signed-certs\"" Aug 05 17:17:24.079037 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.079008 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r"] Aug 05 17:17:24.103381 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.103340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:24.155934 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.155896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-dshm\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.155934 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.155940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-tmp-dir\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.156157 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.156010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc6qd\" (UniqueName: \"kubernetes.io/projected/985d9006-3d07-4190-8059-39ecdc408385-kube-api-access-mc6qd\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.156157 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.156087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-model-cache\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.156157 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.156133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-home\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.156276 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.156188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/985d9006-3d07-4190-8059-39ecdc408385-tls-certs\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.257463 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.257415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-model-cache\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.257668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.257485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-home\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.257668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.257528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/985d9006-3d07-4190-8059-39ecdc408385-tls-certs\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.257668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.257556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-dshm\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.257668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.257592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-tmp-dir\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.257668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.257640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6qd\" (UniqueName: \"kubernetes.io/projected/985d9006-3d07-4190-8059-39ecdc408385-kube-api-access-mc6qd\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.257987 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.257855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-model-cache\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.257987 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.257913 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-home\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.258079 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.258004 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-tmp-dir\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.260267 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.260235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-dshm\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.260527 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.260507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/985d9006-3d07-4190-8059-39ecdc408385-tls-certs\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.267808 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.267769 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc6qd\" (UniqueName: \"kubernetes.io/projected/985d9006-3d07-4190-8059-39ecdc408385-kube-api-access-mc6qd\") pod \"rate-test-model-92434d-kserve-766564c9d7-f769r\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.373221 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.373116 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:24.525231 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:24.525203 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r"] Aug 05 17:17:24.527512 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:17:24.527484 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod985d9006_3d07_4190_8059_39ecdc408385.slice/crio-56b1614392d3fc555ca8fb133e9860be6c8fef8f46266fbc7193e34a9807fa70 WatchSource:0}: Error finding container 56b1614392d3fc555ca8fb133e9860be6c8fef8f46266fbc7193e34a9807fa70: Status 404 returned error can't find the container with id 56b1614392d3fc555ca8fb133e9860be6c8fef8f46266fbc7193e34a9807fa70 Aug 05 17:17:25.352210 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:25.352176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" event={"ID":"985d9006-3d07-4190-8059-39ecdc408385","Type":"ContainerStarted","Data":"4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd"} Aug 05 17:17:25.352210 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:25.352218 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" event={"ID":"985d9006-3d07-4190-8059-39ecdc408385","Type":"ContainerStarted","Data":"56b1614392d3fc555ca8fb133e9860be6c8fef8f46266fbc7193e34a9807fa70"} Aug 05 17:17:25.373923 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:25.373868 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" podStartSLOduration=1.373846979 podStartE2EDuration="1.373846979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:17:25.371759064 +0000 UTC m=+3268.731010911" watchObservedRunningTime="2026-08-05 17:17:25.373846979 +0000 UTC m=+3268.733098850" Aug 05 17:17:26.970265 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:26.970221 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:17:30.585110 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.585048 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" podUID="dcdbe929-2a37-4231-abbe-b086cfd6677b" containerName="main" containerID="cri-o://776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2" gracePeriod=45 Aug 05 17:17:30.842626 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.842557 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:17:30.921316 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921276 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-home\") pod \"dcdbe929-2a37-4231-abbe-b086cfd6677b\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " Aug 05 17:17:30.921316 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921327 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-model-cache\") pod \"dcdbe929-2a37-4231-abbe-b086cfd6677b\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " Aug 05 17:17:30.921534 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921349 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctbz7\" (UniqueName: \"kubernetes.io/projected/dcdbe929-2a37-4231-abbe-b086cfd6677b-kube-api-access-ctbz7\") pod \"dcdbe929-2a37-4231-abbe-b086cfd6677b\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " Aug 05 17:17:30.921534 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921399 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-tmp-dir\") pod \"dcdbe929-2a37-4231-abbe-b086cfd6677b\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " Aug 05 17:17:30.921534 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921462 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdbe929-2a37-4231-abbe-b086cfd6677b-tls-certs\") pod \"dcdbe929-2a37-4231-abbe-b086cfd6677b\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " Aug 05 17:17:30.921534 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921484 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-dshm\") pod \"dcdbe929-2a37-4231-abbe-b086cfd6677b\" (UID: \"dcdbe929-2a37-4231-abbe-b086cfd6677b\") " Aug 05 17:17:30.921752 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921569 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-home" (OuterVolumeSpecName: "home") pod "dcdbe929-2a37-4231-abbe-b086cfd6677b" (UID: "dcdbe929-2a37-4231-abbe-b086cfd6677b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:30.921752 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921602 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-model-cache" (OuterVolumeSpecName: "model-cache") pod "dcdbe929-2a37-4231-abbe-b086cfd6677b" (UID: "dcdbe929-2a37-4231-abbe-b086cfd6677b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:30.921902 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921750 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dcdbe929-2a37-4231-abbe-b086cfd6677b" (UID: "dcdbe929-2a37-4231-abbe-b086cfd6677b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:30.921902 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921835 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:30.921902 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921849 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:30.921902 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.921864 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:30.923779 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.923745 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-dshm" (OuterVolumeSpecName: "dshm") pod "dcdbe929-2a37-4231-abbe-b086cfd6677b" (UID: "dcdbe929-2a37-4231-abbe-b086cfd6677b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:30.923779 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.923756 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcdbe929-2a37-4231-abbe-b086cfd6677b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dcdbe929-2a37-4231-abbe-b086cfd6677b" (UID: "dcdbe929-2a37-4231-abbe-b086cfd6677b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:17:30.923942 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:30.923785 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdbe929-2a37-4231-abbe-b086cfd6677b-kube-api-access-ctbz7" (OuterVolumeSpecName: "kube-api-access-ctbz7") pod "dcdbe929-2a37-4231-abbe-b086cfd6677b" (UID: "dcdbe929-2a37-4231-abbe-b086cfd6677b"). InnerVolumeSpecName "kube-api-access-ctbz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:17:31.023021 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.022962 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctbz7\" (UniqueName: \"kubernetes.io/projected/dcdbe929-2a37-4231-abbe-b086cfd6677b-kube-api-access-ctbz7\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:31.023021 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.023011 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdbe929-2a37-4231-abbe-b086cfd6677b-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:31.023021 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.023023 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdbe929-2a37-4231-abbe-b086cfd6677b-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:31.382989 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.382898 2575 generic.go:356] "Generic (PLEG): container finished" podID="dcdbe929-2a37-4231-abbe-b086cfd6677b" containerID="776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2" exitCode=0 Aug 05 17:17:31.382989 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.382955 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" Aug 05 17:17:31.382989 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.382977 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" event={"ID":"dcdbe929-2a37-4231-abbe-b086cfd6677b","Type":"ContainerDied","Data":"776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2"} Aug 05 17:17:31.383227 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.383028 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p" event={"ID":"dcdbe929-2a37-4231-abbe-b086cfd6677b","Type":"ContainerDied","Data":"6a899b6e17657fbe6a48502ed3cd461aa2382a65233d2413570c94e1d9b4f1be"} Aug 05 17:17:31.383227 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.383042 2575 scope.go:117] "RemoveContainer" containerID="776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2" Aug 05 17:17:31.392285 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.392270 2575 scope.go:117] "RemoveContainer" containerID="776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2" Aug 05 17:17:31.392620 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:17:31.392603 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2\": container with ID starting with 776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2 not found: ID does not exist" containerID="776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2" Aug 05 17:17:31.392665 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.392631 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2"} err="failed to get container status \"776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2\": rpc error: code = NotFound desc = could not find container \"776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2\": container with ID starting with 776af653bcb99a3c6a95c7008d1cc61bc2fdb774b3ec24cfc461452d1d4421e2 not found: ID does not exist" Aug 05 17:17:31.401869 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.401835 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p"] Aug 05 17:17:31.405452 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.405426 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/sub-test-model-92434d-kserve-768cbd75bc-lbx8p"] Aug 05 17:17:31.556505 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.556469 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" podUID="f33d449e-a952-425e-bf8a-59651551f2c0" containerName="main" containerID="cri-o://cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748" gracePeriod=45 Aug 05 17:17:31.807853 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.807826 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:17:31.930417 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.930311 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79gh7\" (UniqueName: \"kubernetes.io/projected/f33d449e-a952-425e-bf8a-59651551f2c0-kube-api-access-79gh7\") pod \"f33d449e-a952-425e-bf8a-59651551f2c0\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " Aug 05 17:17:31.930417 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.930379 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-dshm\") pod \"f33d449e-a952-425e-bf8a-59651551f2c0\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " Aug 05 17:17:31.930417 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.930401 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f33d449e-a952-425e-bf8a-59651551f2c0-tls-certs\") pod \"f33d449e-a952-425e-bf8a-59651551f2c0\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " Aug 05 17:17:31.930708 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.930426 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-model-cache\") pod \"f33d449e-a952-425e-bf8a-59651551f2c0\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " Aug 05 17:17:31.930708 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.930447 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-tmp-dir\") pod \"f33d449e-a952-425e-bf8a-59651551f2c0\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " Aug 05 17:17:31.930708 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.930505 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-home\") pod \"f33d449e-a952-425e-bf8a-59651551f2c0\" (UID: \"f33d449e-a952-425e-bf8a-59651551f2c0\") " Aug 05 17:17:31.930910 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.930729 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f33d449e-a952-425e-bf8a-59651551f2c0" (UID: "f33d449e-a952-425e-bf8a-59651551f2c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:31.930910 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.930748 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f33d449e-a952-425e-bf8a-59651551f2c0" (UID: "f33d449e-a952-425e-bf8a-59651551f2c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:31.930910 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.930892 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-home" (OuterVolumeSpecName: "home") pod "f33d449e-a952-425e-bf8a-59651551f2c0" (UID: "f33d449e-a952-425e-bf8a-59651551f2c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:31.932777 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.932755 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33d449e-a952-425e-bf8a-59651551f2c0-kube-api-access-79gh7" (OuterVolumeSpecName: "kube-api-access-79gh7") pod "f33d449e-a952-425e-bf8a-59651551f2c0" (UID: "f33d449e-a952-425e-bf8a-59651551f2c0"). InnerVolumeSpecName "kube-api-access-79gh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:17:31.932884 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.932774 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-dshm" (OuterVolumeSpecName: "dshm") pod "f33d449e-a952-425e-bf8a-59651551f2c0" (UID: "f33d449e-a952-425e-bf8a-59651551f2c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:31.932884 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:31.932852 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f33d449e-a952-425e-bf8a-59651551f2c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f33d449e-a952-425e-bf8a-59651551f2c0" (UID: "f33d449e-a952-425e-bf8a-59651551f2c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:17:32.032218 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.032173 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f33d449e-a952-425e-bf8a-59651551f2c0-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:32.032218 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.032208 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:32.032218 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.032218 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:32.032218 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.032226 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:32.032218 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.032234 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gh7\" (UniqueName: \"kubernetes.io/projected/f33d449e-a952-425e-bf8a-59651551f2c0-kube-api-access-79gh7\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:32.032526 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.032243 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f33d449e-a952-425e-bf8a-59651551f2c0-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:17:32.390275 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.390242 2575 generic.go:356] "Generic (PLEG): container finished" podID="f33d449e-a952-425e-bf8a-59651551f2c0" containerID="cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748" exitCode=0 Aug 05 17:17:32.390433 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.390284 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" Aug 05 17:17:32.390433 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.390324 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" event={"ID":"f33d449e-a952-425e-bf8a-59651551f2c0","Type":"ContainerDied","Data":"cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748"} Aug 05 17:17:32.390433 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.390367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f" event={"ID":"f33d449e-a952-425e-bf8a-59651551f2c0","Type":"ContainerDied","Data":"edee76b6f645bae11e4e575cd42d12087749e8c37cf02ce9601a3e430612d957"} Aug 05 17:17:32.390433 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.390385 2575 scope.go:117] "RemoveContainer" containerID="cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748" Aug 05 17:17:32.400000 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.399973 2575 scope.go:117] "RemoveContainer" containerID="cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748" Aug 05 17:17:32.400316 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:17:32.400298 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748\": container with ID starting with cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748 not found: ID does not exist" containerID="cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748" Aug 05 17:17:32.400394 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.400327 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748"} err="failed to get container status \"cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748\": rpc error: code = NotFound desc = could not find container \"cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748\": container with ID starting with cf6cb51e4f54204770d22d178033393bfbe4ddd26bd991cc63290208b3f4f748 not found: ID does not exist" Aug 05 17:17:32.413607 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.413570 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f"] Aug 05 17:17:32.417733 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:32.417700 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/sub-test-model-57997e-kserve-967d5b8dc-hnx4f"] Aug 05 17:17:33.121304 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:33.121268 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcdbe929-2a37-4231-abbe-b086cfd6677b" path="/var/lib/kubelet/pods/dcdbe929-2a37-4231-abbe-b086cfd6677b/volumes" Aug 05 17:17:33.121695 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:33.121674 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f33d449e-a952-425e-bf8a-59651551f2c0" path="/var/lib/kubelet/pods/f33d449e-a952-425e-bf8a-59651551f2c0/volumes" Aug 05 17:17:34.373467 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:34.373422 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:34.373944 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:34.373485 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:34.386110 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:34.386080 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:34.413869 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:34.413832 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:17:57.326691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:57.326648 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 17:17:57.335406 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:17:57.335381 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 17:18:07.146488 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.146450 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6"] Aug 05 17:18:07.147012 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.146849 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f33d449e-a952-425e-bf8a-59651551f2c0" containerName="main" Aug 05 17:18:07.147012 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.146862 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f33d449e-a952-425e-bf8a-59651551f2c0" containerName="main" Aug 05 17:18:07.147012 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.146871 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcdbe929-2a37-4231-abbe-b086cfd6677b" containerName="main" Aug 05 17:18:07.147012 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.146877 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcdbe929-2a37-4231-abbe-b086cfd6677b" containerName="main" Aug 05 17:18:07.147012 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.146935 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f33d449e-a952-425e-bf8a-59651551f2c0" containerName="main" Aug 05 17:18:07.147012 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.146949 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcdbe929-2a37-4231-abbe-b086cfd6677b" containerName="main" Aug 05 17:18:07.151322 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.151301 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.154185 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.154163 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"kube-root-ca.crt\"" Aug 05 17:18:07.155497 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.155475 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"default-dockercfg-klcgc\"" Aug 05 17:18:07.155634 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.155477 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"openshift-service-ca.crt\"" Aug 05 17:18:07.155634 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.155478 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-57997e\"/\"rate-test-model-57997e-kserve-self-signed-certs\"" Aug 05 17:18:07.164020 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.163994 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6"] Aug 05 17:18:07.237601 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.237561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-tmp-dir\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.237831 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.237752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlql9\" (UniqueName: \"kubernetes.io/projected/084877f6-5174-4a00-b0b2-bf88a95e1561-kube-api-access-mlql9\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.237921 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.237833 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-home\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.237921 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.237886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-dshm\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.238034 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.237970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084877f6-5174-4a00-b0b2-bf88a95e1561-tls-certs\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.238034 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.238031 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-model-cache\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.338742 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.338702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-tmp-dir\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.338943 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.338853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlql9\" (UniqueName: \"kubernetes.io/projected/084877f6-5174-4a00-b0b2-bf88a95e1561-kube-api-access-mlql9\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.338943 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.338892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-home\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.338943 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.338930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-dshm\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.339087 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.338969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084877f6-5174-4a00-b0b2-bf88a95e1561-tls-certs\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.339153 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.339073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-model-cache\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.339153 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.339122 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-tmp-dir\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.339344 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.339324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-home\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.339396 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.339344 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-model-cache\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.341728 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.341708 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-dshm\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.341728 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.341714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084877f6-5174-4a00-b0b2-bf88a95e1561-tls-certs\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.348337 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.348311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlql9\" (UniqueName: \"kubernetes.io/projected/084877f6-5174-4a00-b0b2-bf88a95e1561-kube-api-access-mlql9\") pod \"rate-test-model-57997e-kserve-7cbd9548c-ssrv6\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.461171 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.461075 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:07.601345 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.601293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6"] Aug 05 17:18:07.603936 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:18:07.603901 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod084877f6_5174_4a00_b0b2_bf88a95e1561.slice/crio-d654dba07c66b0434287f629fb111c1babb8c4deaa4a32ee8cc2dd4539dae0ff WatchSource:0}: Error finding container d654dba07c66b0434287f629fb111c1babb8c4deaa4a32ee8cc2dd4539dae0ff: Status 404 returned error can't find the container with id d654dba07c66b0434287f629fb111c1babb8c4deaa4a32ee8cc2dd4539dae0ff Aug 05 17:18:07.978685 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:07.978646 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:08.577864 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:08.577826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" event={"ID":"084877f6-5174-4a00-b0b2-bf88a95e1561","Type":"ContainerStarted","Data":"be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c"} Aug 05 17:18:08.578314 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:08.577875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" event={"ID":"084877f6-5174-4a00-b0b2-bf88a95e1561","Type":"ContainerStarted","Data":"d654dba07c66b0434287f629fb111c1babb8c4deaa4a32ee8cc2dd4539dae0ff"} Aug 05 17:18:08.596009 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:08.595956 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" podStartSLOduration=1.595939643 podStartE2EDuration="1.595939643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:18:08.595368833 +0000 UTC m=+3311.954620676" watchObservedRunningTime="2026-08-05 17:18:08.595939643 +0000 UTC m=+3311.955191488" Aug 05 17:18:09.849657 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:09.849626 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:18:17.461659 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:17.461617 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:17.461659 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:17.461670 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:17.474815 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:17.474765 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:17.633877 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:17.633849 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:18:45.054376 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:45.054321 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:18:55.161068 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:18:55.161030 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:19:10.972232 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:10.972195 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:19:17.306939 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:17.306906 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r"] Aug 05 17:19:17.334493 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:17.334454 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:18.143421 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:18.143378 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6"] Aug 05 17:19:18.166862 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:18.166825 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:19.245917 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.245888 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7748-140073044142896"] Aug 05 17:19:19.249348 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.249328 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044142896" Aug 05 17:19:19.252553 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.252527 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7c4nd\"" Aug 05 17:19:19.259607 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.259582 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-140073044142896"] Aug 05 17:19:19.348222 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.348179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzspb\" (UniqueName: \"kubernetes.io/projected/2dbea110-cec8-40ad-a18b-17895854e19b-kube-api-access-hzspb\") pod \"test-curl-7748-140073044142896\" (UID: \"2dbea110-cec8-40ad-a18b-17895854e19b\") " pod="opendatahub/test-curl-7748-140073044142896" Aug 05 17:19:19.449737 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.449695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzspb\" (UniqueName: \"kubernetes.io/projected/2dbea110-cec8-40ad-a18b-17895854e19b-kube-api-access-hzspb\") pod \"test-curl-7748-140073044142896\" (UID: \"2dbea110-cec8-40ad-a18b-17895854e19b\") " pod="opendatahub/test-curl-7748-140073044142896" Aug 05 17:19:19.464586 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.464552 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzspb\" (UniqueName: \"kubernetes.io/projected/2dbea110-cec8-40ad-a18b-17895854e19b-kube-api-access-hzspb\") pod \"test-curl-7748-140073044142896\" (UID: \"2dbea110-cec8-40ad-a18b-17895854e19b\") " pod="opendatahub/test-curl-7748-140073044142896" Aug 05 17:19:19.559856 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.559751 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044142896" Aug 05 17:19:19.703579 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.703544 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-140073044142896"] Aug 05 17:19:19.706590 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:19:19.706555 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dbea110_cec8_40ad_a18b_17895854e19b.slice/crio-fd635a313db8176c8e266b468cfe16abf4288ea1a5577229f6dc1b20ee86b9db WatchSource:0}: Error finding container fd635a313db8176c8e266b468cfe16abf4288ea1a5577229f6dc1b20ee86b9db: Status 404 returned error can't find the container with id fd635a313db8176c8e266b468cfe16abf4288ea1a5577229f6dc1b20ee86b9db Aug 05 17:19:19.707905 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.707890 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:19:19.931366 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:19.931277 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-140073044142896" event={"ID":"2dbea110-cec8-40ad-a18b-17895854e19b","Type":"ContainerStarted","Data":"fd635a313db8176c8e266b468cfe16abf4288ea1a5577229f6dc1b20ee86b9db"} Aug 05 17:19:20.937369 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:20.937335 2575 generic.go:356] "Generic (PLEG): container finished" podID="2dbea110-cec8-40ad-a18b-17895854e19b" containerID="02f776a2cbf1c500dc149877e36ceae352f3c425e7e8400beecd66cfd81008e6" exitCode=0 Aug 05 17:19:20.937762 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:20.937386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-140073044142896" event={"ID":"2dbea110-cec8-40ad-a18b-17895854e19b","Type":"ContainerDied","Data":"02f776a2cbf1c500dc149877e36ceae352f3c425e7e8400beecd66cfd81008e6"} Aug 05 17:19:22.081058 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.081030 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044142896" Aug 05 17:19:22.102882 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.102854 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7748-140073044142896_2dbea110-cec8-40ad-a18b-17895854e19b/test-curl-7748-140073044142896/0.log" Aug 05 17:19:22.147953 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.147916 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7748-140073044142896"] Aug 05 17:19:22.152076 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.152049 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7748-140073044142896"] Aug 05 17:19:22.178773 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.178732 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzspb\" (UniqueName: \"kubernetes.io/projected/2dbea110-cec8-40ad-a18b-17895854e19b-kube-api-access-hzspb\") pod \"2dbea110-cec8-40ad-a18b-17895854e19b\" (UID: \"2dbea110-cec8-40ad-a18b-17895854e19b\") " Aug 05 17:19:22.181157 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.181133 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbea110-cec8-40ad-a18b-17895854e19b-kube-api-access-hzspb" (OuterVolumeSpecName: "kube-api-access-hzspb") pod "2dbea110-cec8-40ad-a18b-17895854e19b" (UID: "2dbea110-cec8-40ad-a18b-17895854e19b"). InnerVolumeSpecName "kube-api-access-hzspb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:19:22.280423 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.280392 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzspb\" (UniqueName: \"kubernetes.io/projected/2dbea110-cec8-40ad-a18b-17895854e19b-kube-api-access-hzspb\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:22.401165 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.401130 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7748-140073044152880"] Aug 05 17:19:22.401558 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.401543 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbea110-cec8-40ad-a18b-17895854e19b" containerName="test-curl-7748-140073044142896" Aug 05 17:19:22.401624 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.401560 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbea110-cec8-40ad-a18b-17895854e19b" containerName="test-curl-7748-140073044142896" Aug 05 17:19:22.401666 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.401627 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbea110-cec8-40ad-a18b-17895854e19b" containerName="test-curl-7748-140073044142896" Aug 05 17:19:22.404591 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.404571 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044152880" Aug 05 17:19:22.411991 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.411678 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-140073044152880"] Aug 05 17:19:22.482468 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.482431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhnzp\" (UniqueName: \"kubernetes.io/projected/0069898f-5665-4006-b976-1591d4c62d3f-kube-api-access-fhnzp\") pod \"test-curl-7748-140073044152880\" (UID: \"0069898f-5665-4006-b976-1591d4c62d3f\") " pod="opendatahub/test-curl-7748-140073044152880" Aug 05 17:19:22.583739 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.583642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhnzp\" (UniqueName: \"kubernetes.io/projected/0069898f-5665-4006-b976-1591d4c62d3f-kube-api-access-fhnzp\") pod \"test-curl-7748-140073044152880\" (UID: \"0069898f-5665-4006-b976-1591d4c62d3f\") " pod="opendatahub/test-curl-7748-140073044152880" Aug 05 17:19:22.593043 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.593017 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhnzp\" (UniqueName: \"kubernetes.io/projected/0069898f-5665-4006-b976-1591d4c62d3f-kube-api-access-fhnzp\") pod \"test-curl-7748-140073044152880\" (UID: \"0069898f-5665-4006-b976-1591d4c62d3f\") " pod="opendatahub/test-curl-7748-140073044152880" Aug 05 17:19:22.714988 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.714935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044152880" Aug 05 17:19:22.948183 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.948099 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044142896" Aug 05 17:19:22.948183 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:22.948132 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd635a313db8176c8e266b468cfe16abf4288ea1a5577229f6dc1b20ee86b9db" Aug 05 17:19:22.959165 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:19:22.959137 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7748-140073044142896\" is forbidden: User \"system:node:ip-10-0-128-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-17.ec2.internal' and this object" podUID="2dbea110-cec8-40ad-a18b-17895854e19b" pod="opendatahub/test-curl-7748-140073044142896" Aug 05 17:19:23.060393 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:23.060351 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-140073044152880"] Aug 05 17:19:23.063647 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:19:23.063617 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0069898f_5665_4006_b976_1591d4c62d3f.slice/crio-bd2475992fdc1e40c45fe3f7d978f471c268eb662fdfeca78b3c8cba9da66b64 WatchSource:0}: Error finding container bd2475992fdc1e40c45fe3f7d978f471c268eb662fdfeca78b3c8cba9da66b64: Status 404 returned error can't find the container with id bd2475992fdc1e40c45fe3f7d978f471c268eb662fdfeca78b3c8cba9da66b64 Aug 05 17:19:23.121083 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:23.121048 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbea110-cec8-40ad-a18b-17895854e19b" path="/var/lib/kubelet/pods/2dbea110-cec8-40ad-a18b-17895854e19b/volumes" Aug 05 17:19:23.258419 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:23.258385 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:19:23.954639 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:23.954600 2575 generic.go:356] "Generic (PLEG): container finished" podID="0069898f-5665-4006-b976-1591d4c62d3f" containerID="6791df316a43d0fb16467f6942648a708541a3f5c76a6767fb23a0bfbb6db871" exitCode=0 Aug 05 17:19:23.954817 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:23.954687 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-140073044152880" event={"ID":"0069898f-5665-4006-b976-1591d4c62d3f","Type":"ContainerDied","Data":"6791df316a43d0fb16467f6942648a708541a3f5c76a6767fb23a0bfbb6db871"} Aug 05 17:19:23.954817 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:23.954733 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-140073044152880" event={"ID":"0069898f-5665-4006-b976-1591d4c62d3f","Type":"ContainerStarted","Data":"bd2475992fdc1e40c45fe3f7d978f471c268eb662fdfeca78b3c8cba9da66b64"} Aug 05 17:19:25.094992 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:25.094961 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044152880" Aug 05 17:19:25.116362 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:25.116329 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7748-140073044152880_0069898f-5665-4006-b976-1591d4c62d3f/test-curl-7748-140073044152880/0.log" Aug 05 17:19:25.141377 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:25.141342 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7748-140073044152880"] Aug 05 17:19:25.143143 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:25.143117 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7748-140073044152880"] Aug 05 17:19:25.210521 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:25.210421 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhnzp\" (UniqueName: \"kubernetes.io/projected/0069898f-5665-4006-b976-1591d4c62d3f-kube-api-access-fhnzp\") pod \"0069898f-5665-4006-b976-1591d4c62d3f\" (UID: \"0069898f-5665-4006-b976-1591d4c62d3f\") " Aug 05 17:19:25.212719 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:25.212690 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0069898f-5665-4006-b976-1591d4c62d3f-kube-api-access-fhnzp" (OuterVolumeSpecName: "kube-api-access-fhnzp") pod "0069898f-5665-4006-b976-1591d4c62d3f" (UID: "0069898f-5665-4006-b976-1591d4c62d3f"). InnerVolumeSpecName "kube-api-access-fhnzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:19:25.311478 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:25.311445 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhnzp\" (UniqueName: \"kubernetes.io/projected/0069898f-5665-4006-b976-1591d4c62d3f-kube-api-access-fhnzp\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:25.965751 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:25.965719 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044152880" Aug 05 17:19:25.965954 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:25.965726 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd2475992fdc1e40c45fe3f7d978f471c268eb662fdfeca78b3c8cba9da66b64" Aug 05 17:19:25.976190 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:19:25.976156 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7748-140073044152880\" is forbidden: User \"system:node:ip-10-0-128-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-17.ec2.internal' and this object" podUID="0069898f-5665-4006-b976-1591d4c62d3f" pod="opendatahub/test-curl-7748-140073044152880" Aug 05 17:19:27.118419 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:19:27.118382 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7748-140073044152880\" is forbidden: User \"system:node:ip-10-0-128-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-17.ec2.internal' and this object" podUID="0069898f-5665-4006-b976-1591d4c62d3f" pod="opendatahub/test-curl-7748-140073044152880" Aug 05 17:19:27.119961 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:27.119941 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0069898f-5665-4006-b976-1591d4c62d3f" path="/var/lib/kubelet/pods/0069898f-5665-4006-b976-1591d4c62d3f/volumes" Aug 05 17:19:32.326102 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.326043 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" podUID="985d9006-3d07-4190-8059-39ecdc408385" containerName="main" containerID="cri-o://4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd" gracePeriod=45 Aug 05 17:19:32.578259 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.578195 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:19:32.679097 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679063 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-dshm\") pod \"985d9006-3d07-4190-8059-39ecdc408385\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " Aug 05 17:19:32.679276 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679120 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc6qd\" (UniqueName: \"kubernetes.io/projected/985d9006-3d07-4190-8059-39ecdc408385-kube-api-access-mc6qd\") pod \"985d9006-3d07-4190-8059-39ecdc408385\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " Aug 05 17:19:32.679276 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679153 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-home\") pod \"985d9006-3d07-4190-8059-39ecdc408385\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " Aug 05 17:19:32.679276 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679171 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/985d9006-3d07-4190-8059-39ecdc408385-tls-certs\") pod \"985d9006-3d07-4190-8059-39ecdc408385\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " Aug 05 17:19:32.679276 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679213 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-tmp-dir\") pod \"985d9006-3d07-4190-8059-39ecdc408385\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " Aug 05 17:19:32.679276 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679230 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-model-cache\") pod \"985d9006-3d07-4190-8059-39ecdc408385\" (UID: \"985d9006-3d07-4190-8059-39ecdc408385\") " Aug 05 17:19:32.679518 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679471 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-home" (OuterVolumeSpecName: "home") pod "985d9006-3d07-4190-8059-39ecdc408385" (UID: "985d9006-3d07-4190-8059-39ecdc408385"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:19:32.679518 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679480 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "985d9006-3d07-4190-8059-39ecdc408385" (UID: "985d9006-3d07-4190-8059-39ecdc408385"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:19:32.679617 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679564 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-model-cache" (OuterVolumeSpecName: "model-cache") pod "985d9006-3d07-4190-8059-39ecdc408385" (UID: "985d9006-3d07-4190-8059-39ecdc408385"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:19:32.679668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679628 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:32.679668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.679640 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:32.681491 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.681467 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985d9006-3d07-4190-8059-39ecdc408385-kube-api-access-mc6qd" (OuterVolumeSpecName: "kube-api-access-mc6qd") pod "985d9006-3d07-4190-8059-39ecdc408385" (UID: "985d9006-3d07-4190-8059-39ecdc408385"). InnerVolumeSpecName "kube-api-access-mc6qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:19:32.681848 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.681827 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985d9006-3d07-4190-8059-39ecdc408385-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "985d9006-3d07-4190-8059-39ecdc408385" (UID: "985d9006-3d07-4190-8059-39ecdc408385"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:19:32.681848 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.681839 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-dshm" (OuterVolumeSpecName: "dshm") pod "985d9006-3d07-4190-8059-39ecdc408385" (UID: "985d9006-3d07-4190-8059-39ecdc408385"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:19:32.780715 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.780673 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:32.780715 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.780707 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc6qd\" (UniqueName: \"kubernetes.io/projected/985d9006-3d07-4190-8059-39ecdc408385-kube-api-access-mc6qd\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:32.780715 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.780718 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/985d9006-3d07-4190-8059-39ecdc408385-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:32.780715 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:32.780729 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/985d9006-3d07-4190-8059-39ecdc408385-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:33.002924 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.002896 2575 generic.go:356] "Generic (PLEG): container finished" podID="985d9006-3d07-4190-8059-39ecdc408385" containerID="4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd" exitCode=0 Aug 05 17:19:33.002924 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.002937 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" Aug 05 17:19:33.003152 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.002971 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" event={"ID":"985d9006-3d07-4190-8059-39ecdc408385","Type":"ContainerDied","Data":"4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd"} Aug 05 17:19:33.003152 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.003014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r" event={"ID":"985d9006-3d07-4190-8059-39ecdc408385","Type":"ContainerDied","Data":"56b1614392d3fc555ca8fb133e9860be6c8fef8f46266fbc7193e34a9807fa70"} Aug 05 17:19:33.003152 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.003032 2575 scope.go:117] "RemoveContainer" containerID="4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd" Aug 05 17:19:33.011935 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.011920 2575 scope.go:117] "RemoveContainer" containerID="4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd" Aug 05 17:19:33.012193 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:19:33.012178 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd\": container with ID starting with 4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd not found: ID does not exist" containerID="4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd" Aug 05 17:19:33.012232 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.012201 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd"} err="failed to get container status \"4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd\": rpc error: code = NotFound desc = could not find container \"4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd\": container with ID starting with 4124908798a8725f57c114d56c1670eac01b52cf4e5b7129b902bb1db359babd not found: ID does not exist" Aug 05 17:19:33.026181 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.026153 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r"] Aug 05 17:19:33.030833 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.030812 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-92434d/rate-test-model-92434d-kserve-766564c9d7-f769r"] Aug 05 17:19:33.120836 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.120782 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="985d9006-3d07-4190-8059-39ecdc408385" path="/var/lib/kubelet/pods/985d9006-3d07-4190-8059-39ecdc408385/volumes" Aug 05 17:19:33.163459 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.163415 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" podUID="084877f6-5174-4a00-b0b2-bf88a95e1561" containerName="main" containerID="cri-o://be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c" gracePeriod=45 Aug 05 17:19:33.411985 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.411960 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:19:33.486687 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.486660 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-dshm\") pod \"084877f6-5174-4a00-b0b2-bf88a95e1561\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " Aug 05 17:19:33.486868 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.486731 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-tmp-dir\") pod \"084877f6-5174-4a00-b0b2-bf88a95e1561\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " Aug 05 17:19:33.486868 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.486778 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-model-cache\") pod \"084877f6-5174-4a00-b0b2-bf88a95e1561\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " Aug 05 17:19:33.486868 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.486829 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlql9\" (UniqueName: \"kubernetes.io/projected/084877f6-5174-4a00-b0b2-bf88a95e1561-kube-api-access-mlql9\") pod \"084877f6-5174-4a00-b0b2-bf88a95e1561\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " Aug 05 17:19:33.486868 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.486848 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084877f6-5174-4a00-b0b2-bf88a95e1561-tls-certs\") pod \"084877f6-5174-4a00-b0b2-bf88a95e1561\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " Aug 05 17:19:33.487018 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.486872 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-home\") pod \"084877f6-5174-4a00-b0b2-bf88a95e1561\" (UID: \"084877f6-5174-4a00-b0b2-bf88a95e1561\") " Aug 05 17:19:33.487076 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.487027 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "084877f6-5174-4a00-b0b2-bf88a95e1561" (UID: "084877f6-5174-4a00-b0b2-bf88a95e1561"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:19:33.487144 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.487123 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-model-cache" (OuterVolumeSpecName: "model-cache") pod "084877f6-5174-4a00-b0b2-bf88a95e1561" (UID: "084877f6-5174-4a00-b0b2-bf88a95e1561"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:19:33.487206 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.487188 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-home" (OuterVolumeSpecName: "home") pod "084877f6-5174-4a00-b0b2-bf88a95e1561" (UID: "084877f6-5174-4a00-b0b2-bf88a95e1561"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:19:33.487253 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.487240 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:33.487287 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.487257 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:33.489034 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.489009 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084877f6-5174-4a00-b0b2-bf88a95e1561-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "084877f6-5174-4a00-b0b2-bf88a95e1561" (UID: "084877f6-5174-4a00-b0b2-bf88a95e1561"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:19:33.489363 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.489339 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-dshm" (OuterVolumeSpecName: "dshm") pod "084877f6-5174-4a00-b0b2-bf88a95e1561" (UID: "084877f6-5174-4a00-b0b2-bf88a95e1561"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:19:33.489473 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.489369 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084877f6-5174-4a00-b0b2-bf88a95e1561-kube-api-access-mlql9" (OuterVolumeSpecName: "kube-api-access-mlql9") pod "084877f6-5174-4a00-b0b2-bf88a95e1561" (UID: "084877f6-5174-4a00-b0b2-bf88a95e1561"). InnerVolumeSpecName "kube-api-access-mlql9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:19:33.588193 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.588108 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlql9\" (UniqueName: \"kubernetes.io/projected/084877f6-5174-4a00-b0b2-bf88a95e1561-kube-api-access-mlql9\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:33.588193 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.588139 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/084877f6-5174-4a00-b0b2-bf88a95e1561-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:33.588193 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.588149 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:33.588193 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:33.588157 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/084877f6-5174-4a00-b0b2-bf88a95e1561-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:34.010355 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.010326 2575 generic.go:356] "Generic (PLEG): container finished" podID="084877f6-5174-4a00-b0b2-bf88a95e1561" containerID="be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c" exitCode=0 Aug 05 17:19:34.010529 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.010369 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" event={"ID":"084877f6-5174-4a00-b0b2-bf88a95e1561","Type":"ContainerDied","Data":"be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c"} Aug 05 17:19:34.010529 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.010371 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" Aug 05 17:19:34.010529 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.010395 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6" event={"ID":"084877f6-5174-4a00-b0b2-bf88a95e1561","Type":"ContainerDied","Data":"d654dba07c66b0434287f629fb111c1babb8c4deaa4a32ee8cc2dd4539dae0ff"} Aug 05 17:19:34.010529 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.010413 2575 scope.go:117] "RemoveContainer" containerID="be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c" Aug 05 17:19:34.019753 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.019730 2575 scope.go:117] "RemoveContainer" containerID="be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c" Aug 05 17:19:34.020037 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:19:34.020019 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c\": container with ID starting with be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c not found: ID does not exist" containerID="be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c" Aug 05 17:19:34.020092 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.020045 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c"} err="failed to get container status \"be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c\": rpc error: code = NotFound desc = could not find container \"be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c\": container with ID starting with be5767fc8f57d2f20b7c0f4f1ae6b270cdc071ca419b1566a0ed3c8c72b2764c not found: ID does not exist" Aug 05 17:19:34.033454 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.033427 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6"] Aug 05 17:19:34.037688 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.037666 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-57997e/rate-test-model-57997e-kserve-7cbd9548c-ssrv6"] Aug 05 17:19:34.571028 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.570904 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7748-140073044054352"] Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.571430 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="985d9006-3d07-4190-8059-39ecdc408385" containerName="main" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.571449 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="985d9006-3d07-4190-8059-39ecdc408385" containerName="main" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.571464 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084877f6-5174-4a00-b0b2-bf88a95e1561" containerName="main" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.571471 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="084877f6-5174-4a00-b0b2-bf88a95e1561" containerName="main" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.571477 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0069898f-5665-4006-b976-1591d4c62d3f" containerName="test-curl-7748-140073044152880" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.571482 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0069898f-5665-4006-b976-1591d4c62d3f" containerName="test-curl-7748-140073044152880" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.571556 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="985d9006-3d07-4190-8059-39ecdc408385" containerName="main" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.571567 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0069898f-5665-4006-b976-1591d4c62d3f" containerName="test-curl-7748-140073044152880" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.571573 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="084877f6-5174-4a00-b0b2-bf88a95e1561" containerName="main" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.576633 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044054352" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.579475 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7c4nd\"" Aug 05 17:19:34.600691 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.579535 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-140073044054352"] Aug 05 17:19:34.698197 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.698158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sspb\" (UniqueName: \"kubernetes.io/projected/57f30921-b876-4108-9b83-13a82ab6d7ed-kube-api-access-8sspb\") pod \"test-curl-7748-140073044054352\" (UID: \"57f30921-b876-4108-9b83-13a82ab6d7ed\") " pod="opendatahub/test-curl-7748-140073044054352" Aug 05 17:19:34.798859 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.798817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sspb\" (UniqueName: \"kubernetes.io/projected/57f30921-b876-4108-9b83-13a82ab6d7ed-kube-api-access-8sspb\") pod \"test-curl-7748-140073044054352\" (UID: \"57f30921-b876-4108-9b83-13a82ab6d7ed\") " pod="opendatahub/test-curl-7748-140073044054352" Aug 05 17:19:34.808368 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.808338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sspb\" (UniqueName: \"kubernetes.io/projected/57f30921-b876-4108-9b83-13a82ab6d7ed-kube-api-access-8sspb\") pod \"test-curl-7748-140073044054352\" (UID: \"57f30921-b876-4108-9b83-13a82ab6d7ed\") " pod="opendatahub/test-curl-7748-140073044054352" Aug 05 17:19:34.888022 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:34.887935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044054352" Aug 05 17:19:35.029000 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:35.028976 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7748-140073044054352"] Aug 05 17:19:35.031032 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:19:35.030999 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57f30921_b876_4108_9b83_13a82ab6d7ed.slice/crio-6bebb8db05e5486dae26e4f2aa87280e5d419e11a3b9fd5536f838bb709fcb3f WatchSource:0}: Error finding container 6bebb8db05e5486dae26e4f2aa87280e5d419e11a3b9fd5536f838bb709fcb3f: Status 404 returned error can't find the container with id 6bebb8db05e5486dae26e4f2aa87280e5d419e11a3b9fd5536f838bb709fcb3f Aug 05 17:19:35.120214 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:35.120189 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="084877f6-5174-4a00-b0b2-bf88a95e1561" path="/var/lib/kubelet/pods/084877f6-5174-4a00-b0b2-bf88a95e1561/volumes" Aug 05 17:19:36.025230 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:36.025194 2575 generic.go:356] "Generic (PLEG): container finished" podID="57f30921-b876-4108-9b83-13a82ab6d7ed" containerID="aa1817d3e232f9ae787c844a7432fbe79e53925f199ec372c9496a6de8946a9f" exitCode=0 Aug 05 17:19:36.025592 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:36.025258 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-140073044054352" event={"ID":"57f30921-b876-4108-9b83-13a82ab6d7ed","Type":"ContainerDied","Data":"aa1817d3e232f9ae787c844a7432fbe79e53925f199ec372c9496a6de8946a9f"} Aug 05 17:19:36.025592 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:36.025287 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7748-140073044054352" event={"ID":"57f30921-b876-4108-9b83-13a82ab6d7ed","Type":"ContainerStarted","Data":"6bebb8db05e5486dae26e4f2aa87280e5d419e11a3b9fd5536f838bb709fcb3f"} Aug 05 17:19:37.171223 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:37.171191 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044054352" Aug 05 17:19:37.192382 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:37.191835 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7748-140073044054352_57f30921-b876-4108-9b83-13a82ab6d7ed/test-curl-7748-140073044054352/0.log" Aug 05 17:19:37.215604 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:37.215568 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7748-140073044054352"] Aug 05 17:19:37.219437 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:37.219413 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7748-140073044054352"] Aug 05 17:19:37.324570 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:37.324479 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sspb\" (UniqueName: \"kubernetes.io/projected/57f30921-b876-4108-9b83-13a82ab6d7ed-kube-api-access-8sspb\") pod \"57f30921-b876-4108-9b83-13a82ab6d7ed\" (UID: \"57f30921-b876-4108-9b83-13a82ab6d7ed\") " Aug 05 17:19:37.326771 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:37.326744 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f30921-b876-4108-9b83-13a82ab6d7ed-kube-api-access-8sspb" (OuterVolumeSpecName: "kube-api-access-8sspb") pod "57f30921-b876-4108-9b83-13a82ab6d7ed" (UID: "57f30921-b876-4108-9b83-13a82ab6d7ed"). InnerVolumeSpecName "kube-api-access-8sspb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:19:37.425247 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:37.425208 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sspb\" (UniqueName: \"kubernetes.io/projected/57f30921-b876-4108-9b83-13a82ab6d7ed-kube-api-access-8sspb\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:19:37.431655 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:37.431627 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:37.730328 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:37.730242 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:38.037116 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:38.037091 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7748-140073044054352" Aug 05 17:19:38.037116 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:38.037123 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bebb8db05e5486dae26e4f2aa87280e5d419e11a3b9fd5536f838bb709fcb3f" Aug 05 17:19:38.055569 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:19:38.055535 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7748-140073044054352\" is forbidden: User \"system:node:ip-10-0-128-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-17.ec2.internal' and this object" podUID="57f30921-b876-4108-9b83-13a82ab6d7ed" pod="opendatahub/test-curl-7748-140073044054352" Aug 05 17:19:38.142053 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:38.142005 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:38.378944 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:38.378846 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:39.122871 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:39.122841 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f30921-b876-4108-9b83-13a82ab6d7ed" path="/var/lib/kubelet/pods/57f30921-b876-4108-9b83-13a82ab6d7ed/volumes" Aug 05 17:19:41.349079 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:41.349031 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:41.448171 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:41.448106 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:42.477371 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:42.477335 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:42.841342 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:42.841308 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:42.997411 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:42.997378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:43.095427 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:43.095315 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:45.266080 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:19:45.266032 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:20:06.349005 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:06.348966 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:07.021930 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.021896 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:07.311136 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.311054 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk"] Aug 05 17:20:07.311455 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.311440 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f30921-b876-4108-9b83-13a82ab6d7ed" containerName="test-curl-7748-140073044054352" Aug 05 17:20:07.311495 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.311458 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f30921-b876-4108-9b83-13a82ab6d7ed" containerName="test-curl-7748-140073044054352" Aug 05 17:20:07.311528 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.311513 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f30921-b876-4108-9b83-13a82ab6d7ed" containerName="test-curl-7748-140073044054352" Aug 05 17:20:07.316920 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.316885 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.319704 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.319681 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ca1c80-openshift-default-dockercfg-s75fd\"" Aug 05 17:20:07.331030 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.331001 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk"] Aug 05 17:20:07.348259 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.348225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:07.382206 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.382170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-certs\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.382657 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.382237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-podinfo\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.382657 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.382286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istiod-ca-cert\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.382657 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.382313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-data\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.382657 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.382377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6h6\" (UniqueName: \"kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-kube-api-access-cr6h6\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.382657 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.382402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-socket\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.382657 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.382472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-token\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.382657 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.382534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-credential-socket\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.382657 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.382570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-envoy\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.483272 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-certs\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.483561 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-podinfo\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.483561 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istiod-ca-cert\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.483693 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-data\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.483693 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr6h6\" (UniqueName: \"kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-kube-api-access-cr6h6\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.483693 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-socket\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.483693 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-token\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.483948 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-credential-socket\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.483948 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-certs\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.483948 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-envoy\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.484096 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.483974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-data\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.484200 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.484172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-socket\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.484200 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.484197 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-credential-socket\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.484381 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.484285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istiod-ca-cert\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.486180 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.486152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-envoy\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.486365 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.486347 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-podinfo\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.492444 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.492417 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-token\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.492544 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.492502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr6h6\" (UniqueName: \"kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-kube-api-access-cr6h6\") pod \"e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.587243 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.587156 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:07.627141 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.627105 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:07.774631 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.774502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk"] Aug 05 17:20:07.777812 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:20:07.777768 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ed34e7_7386_442b_89ed_45e8ac707fd1.slice/crio-2a67ef990d30458b8fb873295eb533672cc6f6d6ef48a25ef4f3149ca281ef53 WatchSource:0}: Error finding container 2a67ef990d30458b8fb873295eb533672cc6f6d6ef48a25ef4f3149ca281ef53: Status 404 returned error can't find the container with id 2a67ef990d30458b8fb873295eb533672cc6f6d6ef48a25ef4f3149ca281ef53 Aug 05 17:20:07.780612 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.780581 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:20:07.780715 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.780645 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:20:07.780715 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:07.780674 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:20:08.187812 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:08.187753 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" event={"ID":"e9ed34e7-7386-442b-89ed-45e8ac707fd1","Type":"ContainerStarted","Data":"cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d"} Aug 05 17:20:08.188016 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:08.187829 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" event={"ID":"e9ed34e7-7386-442b-89ed-45e8ac707fd1","Type":"ContainerStarted","Data":"2a67ef990d30458b8fb873295eb533672cc6f6d6ef48a25ef4f3149ca281ef53"} Aug 05 17:20:08.219916 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:08.219862 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" podStartSLOduration=1.219845147 podStartE2EDuration="1.219845147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:08.21867221 +0000 UTC m=+3431.577924053" watchObservedRunningTime="2026-08-05 17:20:08.219845147 +0000 UTC m=+3431.579096990" Aug 05 17:20:08.628243 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:08.628205 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:08.633439 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:08.633412 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:09.192651 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:09.192618 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:09.193755 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:09.193734 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:10.163907 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:10.163878 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:10.260848 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:10.260765 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:11.372081 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.372045 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8"] Aug 05 17:20:11.376048 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.376027 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:11.379151 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.379122 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 17:20:11.379736 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.379720 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ca1c80-serving-cert\"" Aug 05 17:20:11.379949 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.379936 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8x4pl\"" Aug 05 17:20:11.380031 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.379993 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 17:20:11.387769 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.387741 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8"] Aug 05 17:20:11.417937 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.417886 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:11.420036 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.420011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10665a-0260-4acc-aa23-cc2567131800-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:11.420110 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.420056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56ns\" (UniqueName: \"kubernetes.io/projected/6b10665a-0260-4acc-aa23-cc2567131800-kube-api-access-t56ns\") pod \"maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:11.420158 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.420142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b10665a-0260-4acc-aa23-cc2567131800-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:11.521152 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.521113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10665a-0260-4acc-aa23-cc2567131800-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:11.521338 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.521164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56ns\" (UniqueName: \"kubernetes.io/projected/6b10665a-0260-4acc-aa23-cc2567131800-kube-api-access-t56ns\") pod \"maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:11.521338 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.521211 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b10665a-0260-4acc-aa23-cc2567131800-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:11.521338 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:20:11.521286 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-serving-cert: secret "maas-api-e2e-ipp-b-ca1c80-serving-cert" not found Aug 05 17:20:11.521446 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:20:11.521365 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b10665a-0260-4acc-aa23-cc2567131800-maas-api-tls podName:6b10665a-0260-4acc-aa23-cc2567131800 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:12.02134242 +0000 UTC m=+3435.380594246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b10665a-0260-4acc-aa23-cc2567131800-maas-api-tls") pod "maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" (UID: "6b10665a-0260-4acc-aa23-cc2567131800") : secret "maas-api-e2e-ipp-b-ca1c80-serving-cert" not found Aug 05 17:20:11.521873 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.521856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b10665a-0260-4acc-aa23-cc2567131800-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:11.539098 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.539073 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56ns\" (UniqueName: \"kubernetes.io/projected/6b10665a-0260-4acc-aa23-cc2567131800-kube-api-access-t56ns\") pod \"maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:11.671760 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:11.671663 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:12.006995 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:12.006959 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:12.026524 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:12.026492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10665a-0260-4acc-aa23-cc2567131800-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:12.029104 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:12.029078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10665a-0260-4acc-aa23-cc2567131800-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:12.094269 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:12.094230 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:12.291360 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:12.291266 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:12.429429 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:12.429366 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8"] Aug 05 17:20:12.432099 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:20:12.432071 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b10665a_0260_4acc_aa23_cc2567131800.slice/crio-4c7cc9885818928ef811a9a54122c69755afc912d3d48dfa77a9eb01e620f89c WatchSource:0}: Error finding container 4c7cc9885818928ef811a9a54122c69755afc912d3d48dfa77a9eb01e620f89c: Status 404 returned error can't find the container with id 4c7cc9885818928ef811a9a54122c69755afc912d3d48dfa77a9eb01e620f89c Aug 05 17:20:13.215189 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:13.215155 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" event={"ID":"6b10665a-0260-4acc-aa23-cc2567131800","Type":"ContainerStarted","Data":"4c7cc9885818928ef811a9a54122c69755afc912d3d48dfa77a9eb01e620f89c"} Aug 05 17:20:15.075493 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:15.075454 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:20:15.226524 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:15.226484 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" event={"ID":"6b10665a-0260-4acc-aa23-cc2567131800","Type":"ContainerStarted","Data":"815dd3a269e702e36b730fab0769fdc7e2c0f6f273cad44978f78501a4bddf3a"} Aug 05 17:20:15.226729 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:15.226547 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:15.250477 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:15.250423 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" podStartSLOduration=2.041486274 podStartE2EDuration="4.250408514s" totalImagesPullingTime="2.20892224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:15.247181109 +0000 UTC m=+3438.606432953" watchObservedRunningTime="2026-08-05 17:20:15.250408514 +0000 UTC m=+3438.609660336" Aug 05 17:20:21.237560 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:21.237528 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:20:24.653493 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:24.653455 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": read tcp 10.134.0.2:45810->10.134.0.52:15021: read: connection reset by peer" start-of-body= Aug 05 17:20:24.654069 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:24.653532 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": read tcp 10.134.0.2:45810->10.134.0.52:15021: read: connection reset by peer" Aug 05 17:20:25.276167 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:25.276134 2575 generic.go:356] "Generic (PLEG): container finished" podID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerID="cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d" exitCode=0 Aug 05 17:20:25.276346 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:25.276209 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" event={"ID":"e9ed34e7-7386-442b-89ed-45e8ac707fd1","Type":"ContainerDied","Data":"cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d"} Aug 05 17:20:25.276570 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:25.276557 2575 scope.go:117] "RemoveContainer" containerID="cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d" Aug 05 17:20:25.277695 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:25.277669 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:20:25.277757 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:25.277738 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:20:25.277813 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:25.277773 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:20:26.286530 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:26.286472 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" event={"ID":"e9ed34e7-7386-442b-89ed-45e8ac707fd1","Type":"ContainerStarted","Data":"b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f"} Aug 05 17:20:26.628108 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:26.628008 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:26.628298 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:26.628276 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Aug 05 17:20:26.628374 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:26.628346 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Aug 05 17:20:27.628330 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:27.628292 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Aug 05 17:20:27.628929 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:27.628356 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Aug 05 17:20:28.627672 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:28.627628 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Aug 05 17:20:28.628020 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:28.627693 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Aug 05 17:20:29.628469 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:29.628435 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Aug 05 17:20:29.628951 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:29.628919 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Aug 05 17:20:30.628240 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:30.628199 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Aug 05 17:20:30.628428 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:30.628257 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Aug 05 17:20:31.631526 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:31.631495 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:31.631932 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:31.631773 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:31.632459 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:31.632439 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:20:35.175186 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:35.175144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:51.667329 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.667291 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q"] Aug 05 17:20:51.670700 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.670683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.673892 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.673868 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0b8087\"/\"openshift-service-ca.crt\"" Aug 05 17:20:51.675272 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.675248 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0b8087\"/\"kube-root-ca.crt\"" Aug 05 17:20:51.675272 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.675259 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0b8087\"/\"ipp-route-0b8087-kserve-self-signed-certs\"" Aug 05 17:20:51.675272 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.675268 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0b8087\"/\"default-dockercfg-88ntw\"" Aug 05 17:20:51.689346 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.689315 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q"] Aug 05 17:20:51.759014 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.758981 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:51.825047 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.825009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tls-certs\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.825232 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.825054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8864\" (UniqueName: \"kubernetes.io/projected/6f4a8a4e-2717-4199-a8c2-9da3171e546b-kube-api-access-c8864\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.825232 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.825139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-model-cache\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.825232 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.825197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-home\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.825347 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.825249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-dshm\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.825347 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.825286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tmp-dir\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.926314 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.926210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-model-cache\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.926314 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.926263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-home\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.926558 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.926324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-dshm\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.926558 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.926369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tmp-dir\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.926558 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.926414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tls-certs\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.926558 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.926445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8864\" (UniqueName: \"kubernetes.io/projected/6f4a8a4e-2717-4199-a8c2-9da3171e546b-kube-api-access-c8864\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.926747 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.926574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-model-cache\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.926747 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.926622 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-home\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.926747 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.926692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tmp-dir\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.928658 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.928637 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-dshm\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.928938 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.928918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tls-certs\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.935555 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.935528 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8864\" (UniqueName: \"kubernetes.io/projected/6f4a8a4e-2717-4199-a8c2-9da3171e546b-kube-api-access-c8864\") pod \"ipp-route-0b8087-kserve-768bd4444c-pqf8q\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:51.980475 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:51.980430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:20:52.118591 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:52.118571 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q"] Aug 05 17:20:52.121098 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:20:52.121067 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f4a8a4e_2717_4199_a8c2_9da3171e546b.slice/crio-2cd62b1ffdfb78b28b42b3b85546487bce5d6598bfb4f0f3890096343de29114 WatchSource:0}: Error finding container 2cd62b1ffdfb78b28b42b3b85546487bce5d6598bfb4f0f3890096343de29114: Status 404 returned error can't find the container with id 2cd62b1ffdfb78b28b42b3b85546487bce5d6598bfb4f0f3890096343de29114 Aug 05 17:20:52.427189 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:52.427154 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" event={"ID":"6f4a8a4e-2717-4199-a8c2-9da3171e546b","Type":"ContainerStarted","Data":"763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5"} Aug 05 17:20:52.427189 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:52.427193 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" event={"ID":"6f4a8a4e-2717-4199-a8c2-9da3171e546b","Type":"ContainerStarted","Data":"2cd62b1ffdfb78b28b42b3b85546487bce5d6598bfb4f0f3890096343de29114"} Aug 05 17:20:52.446727 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:52.446625 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" podStartSLOduration=1.446609281 podStartE2EDuration="1.446609281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:52.444209761 +0000 UTC m=+3475.803461604" watchObservedRunningTime="2026-08-05 17:20:52.446609281 +0000 UTC m=+3475.805861125" Aug 05 17:20:53.659853 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:20:53.659812 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:21:01.981592 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:01.981549 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:21:01.982135 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:01.981618 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:21:01.993968 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:01.993939 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:21:02.491228 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:02.491195 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:21:21.246038 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:21.245946 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:21:32.583438 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:32.583399 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:32.998847 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:32.998806 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:33.358281 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:33.358197 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:35.923026 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:35.922986 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:36.020172 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:36.020123 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:37.022827 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.022772 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9"] Aug 05 17:21:37.026682 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.026660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.029571 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.029548 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-32fc6e-serving-cert\"" Aug 05 17:21:37.041131 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.041105 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9"] Aug 05 17:21:37.061414 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.061375 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:37.143944 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.143908 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tfgh\" (UniqueName: \"kubernetes.io/projected/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-kube-api-access-4tfgh\") pod \"maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.144140 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.143989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.144186 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.144153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.245245 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.245199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.245441 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.245293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfgh\" (UniqueName: \"kubernetes.io/projected/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-kube-api-access-4tfgh\") pod \"maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.245441 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.245366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.245564 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:21:37.245549 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-serving-cert: secret "maas-api-e2e-ipp-cleanup-32fc6e-serving-cert" not found Aug 05 17:21:37.245663 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:21:37.245650 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-maas-api-tls podName:d25aad33-d1cb-44eb-9f9f-d787f9284cc7 nodeName:}" failed. No retries permitted until 2026-08-05 17:21:37.745628727 +0000 UTC m=+3521.104880553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" (UID: "d25aad33-d1cb-44eb-9f9f-d787f9284cc7") : secret "maas-api-e2e-ipp-cleanup-32fc6e-serving-cert" not found Aug 05 17:21:37.245983 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.245963 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.256999 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.256978 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tfgh\" (UniqueName: \"kubernetes.io/projected/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-kube-api-access-4tfgh\") pod \"maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.406471 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.406354 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:37.593874 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.593832 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:37.691271 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.691167 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:37.751068 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.751034 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.753929 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.753899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:37.936976 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:37.936934 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:38.085225 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:38.085193 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9"] Aug 05 17:21:38.087306 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:21:38.087272 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25aad33_d1cb_44eb_9f9f_d787f9284cc7.slice/crio-a2e5ffe1f12406ae56a25118d9c8e1c128ab4fd993756faa31514b2f6a43ffea WatchSource:0}: Error finding container a2e5ffe1f12406ae56a25118d9c8e1c128ab4fd993756faa31514b2f6a43ffea: Status 404 returned error can't find the container with id a2e5ffe1f12406ae56a25118d9c8e1c128ab4fd993756faa31514b2f6a43ffea Aug 05 17:21:38.665662 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:38.665626 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" event={"ID":"d25aad33-d1cb-44eb-9f9f-d787f9284cc7","Type":"ContainerStarted","Data":"bc9d8c02ee5141084034bc0785151e25df3dcebca77dffcf1c69c8020bf746af"} Aug 05 17:21:38.665876 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:38.665684 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" event={"ID":"d25aad33-d1cb-44eb-9f9f-d787f9284cc7","Type":"ContainerStarted","Data":"a2e5ffe1f12406ae56a25118d9c8e1c128ab4fd993756faa31514b2f6a43ffea"} Aug 05 17:21:38.665876 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:38.665713 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:21:38.686026 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:38.685970 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" podStartSLOduration=1.27835876 podStartE2EDuration="1.685949166s" totalImagesPullingTime="407.590406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:38.68325512 +0000 UTC m=+3522.042506973" watchObservedRunningTime="2026-08-05 17:21:38.685949166 +0000 UTC m=+3522.045201012" Aug 05 17:21:43.056646 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:43.056599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:21:44.680093 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:21:44.680060 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:22:00.904665 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:00.904622 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:07.577699 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.577665 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9"] Aug 05 17:22:07.578293 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.577914 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" podUID="d25aad33-d1cb-44eb-9f9f-d787f9284cc7" containerName="maas-api" containerID="cri-o://bc9d8c02ee5141084034bc0785151e25df3dcebca77dffcf1c69c8020bf746af" gracePeriod=30 Aug 05 17:22:07.609511 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.609463 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:07.687411 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.687374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:07.756599 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.756559 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:07.824983 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.824856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:07.831816 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.831703 2575 generic.go:356] "Generic (PLEG): container finished" podID="d25aad33-d1cb-44eb-9f9f-d787f9284cc7" containerID="bc9d8c02ee5141084034bc0785151e25df3dcebca77dffcf1c69c8020bf746af" exitCode=0 Aug 05 17:22:07.831974 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.831863 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" event={"ID":"d25aad33-d1cb-44eb-9f9f-d787f9284cc7","Type":"ContainerDied","Data":"bc9d8c02ee5141084034bc0785151e25df3dcebca77dffcf1c69c8020bf746af"} Aug 05 17:22:07.831974 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.831916 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" event={"ID":"d25aad33-d1cb-44eb-9f9f-d787f9284cc7","Type":"ContainerDied","Data":"a2e5ffe1f12406ae56a25118d9c8e1c128ab4fd993756faa31514b2f6a43ffea"} Aug 05 17:22:07.831974 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.831941 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2e5ffe1f12406ae56a25118d9c8e1c128ab4fd993756faa31514b2f6a43ffea" Aug 05 17:22:07.840329 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.840304 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:22:07.956417 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.956386 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-openshift-service-ca\") pod \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " Aug 05 17:22:07.956583 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.956452 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tfgh\" (UniqueName: \"kubernetes.io/projected/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-kube-api-access-4tfgh\") pod \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " Aug 05 17:22:07.956651 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.956628 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-maas-api-tls\") pod \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\" (UID: \"d25aad33-d1cb-44eb-9f9f-d787f9284cc7\") " Aug 05 17:22:07.956727 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.956705 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d25aad33-d1cb-44eb-9f9f-d787f9284cc7" (UID: "d25aad33-d1cb-44eb-9f9f-d787f9284cc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:07.957098 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.957075 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-openshift-service-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:22:07.958829 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.958768 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-kube-api-access-4tfgh" (OuterVolumeSpecName: "kube-api-access-4tfgh") pod "d25aad33-d1cb-44eb-9f9f-d787f9284cc7" (UID: "d25aad33-d1cb-44eb-9f9f-d787f9284cc7"). InnerVolumeSpecName "kube-api-access-4tfgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:07.958926 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:07.958845 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d25aad33-d1cb-44eb-9f9f-d787f9284cc7" (UID: "d25aad33-d1cb-44eb-9f9f-d787f9284cc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:08.057808 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:08.057761 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tfgh\" (UniqueName: \"kubernetes.io/projected/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-kube-api-access-4tfgh\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:22:08.057808 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:08.057805 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25aad33-d1cb-44eb-9f9f-d787f9284cc7-maas-api-tls\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:22:08.217056 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:08.216960 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:08.837610 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:08.837574 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9" Aug 05 17:22:08.858985 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:08.858947 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9"] Aug 05 17:22:08.865127 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:08.865091 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-32fc6e-77dfcff4c9-85hf9"] Aug 05 17:22:09.120257 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:09.120177 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25aad33-d1cb-44eb-9f9f-d787f9284cc7" path="/var/lib/kubelet/pods/d25aad33-d1cb-44eb-9f9f-d787f9284cc7/volumes" Aug 05 17:22:10.759490 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:10.759453 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:22:14.107486 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:14.107450 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:14.207988 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:14.207948 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:14.370820 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:14.370710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:14.611325 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:14.611278 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:18.061266 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.061230 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc"] Aug 05 17:22:18.061644 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.061617 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25aad33-d1cb-44eb-9f9f-d787f9284cc7" containerName="maas-api" Aug 05 17:22:18.061644 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.061628 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25aad33-d1cb-44eb-9f9f-d787f9284cc7" containerName="maas-api" Aug 05 17:22:18.061718 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.061692 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25aad33-d1cb-44eb-9f9f-d787f9284cc7" containerName="maas-api" Aug 05 17:22:18.064776 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.064759 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" Aug 05 17:22:18.067962 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.067939 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68wxh\"" Aug 05 17:22:18.073701 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.073677 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc"] Aug 05 17:22:18.153548 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.153506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc\" (UID: \"1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" Aug 05 17:22:18.254912 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.254871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc\" (UID: \"1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" Aug 05 17:22:18.255541 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.255519 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc\" (UID: \"1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" Aug 05 17:22:18.377090 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.376988 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" Aug 05 17:22:18.521573 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.521543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc"] Aug 05 17:22:18.523527 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:22:18.523500 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b69cf03_7d4c_43cf_80e1_3d7f7a9466ec.slice/crio-d3570e63f225fc13b6d5d722de6ef432196673713b2c93bb384ed7b1cfb92b75 WatchSource:0}: Error finding container d3570e63f225fc13b6d5d722de6ef432196673713b2c93bb384ed7b1cfb92b75: Status 404 returned error can't find the container with id d3570e63f225fc13b6d5d722de6ef432196673713b2c93bb384ed7b1cfb92b75 Aug 05 17:22:18.892065 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.892028 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" event={"ID":"1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec","Type":"ContainerStarted","Data":"b7e922c4106ede07fd2ee0ecdda86f9db8a233d0ffaa3a45aaa8c30653931ce4"} Aug 05 17:22:18.892268 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.892082 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" event={"ID":"1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec","Type":"ContainerStarted","Data":"d3570e63f225fc13b6d5d722de6ef432196673713b2c93bb384ed7b1cfb92b75"} Aug 05 17:22:18.909375 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:18.909316 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" podStartSLOduration=0.909297169 podStartE2EDuration="909.297169ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:18.907825794 +0000 UTC m=+3562.267077632" watchObservedRunningTime="2026-08-05 17:22:18.909297169 +0000 UTC m=+3562.268549051" Aug 05 17:22:19.898931 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:19.898894 2575 generic.go:356] "Generic (PLEG): container finished" podID="1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec" containerID="b7e922c4106ede07fd2ee0ecdda86f9db8a233d0ffaa3a45aaa8c30653931ce4" exitCode=0 Aug 05 17:22:19.899318 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:19.898981 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" event={"ID":"1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec","Type":"ContainerDied","Data":"b7e922c4106ede07fd2ee0ecdda86f9db8a233d0ffaa3a45aaa8c30653931ce4"} Aug 05 17:22:21.033351 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:21.033320 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" Aug 05 17:22:21.185832 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:21.185714 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec-maas-api-service-ca\") pod \"1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec\" (UID: \"1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec\") " Aug 05 17:22:21.186150 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:21.186130 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec" (UID: "1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:21.287585 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:21.287548 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec-maas-api-service-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:22:21.909728 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:21.909695 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" Aug 05 17:22:21.909728 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:21.909717 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc" event={"ID":"1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec","Type":"ContainerDied","Data":"d3570e63f225fc13b6d5d722de6ef432196673713b2c93bb384ed7b1cfb92b75"} Aug 05 17:22:21.909957 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:21.909765 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3570e63f225fc13b6d5d722de6ef432196673713b2c93bb384ed7b1cfb92b75" Aug 05 17:22:25.449623 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:25.449582 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:22:26.746665 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:26.746626 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:26.848156 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:26.848119 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:27.239843 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:27.239724 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:27.343387 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:27.343353 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:27.442546 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:27.442505 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:27.443383 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:27.443349 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc"] Aug 05 17:22:27.444993 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:27.444970 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0b8087-wrbsc"] Aug 05 17:22:27.508778 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:27.508739 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:27.790238 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:27.790145 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:29.120742 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:29.120705 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec" path="/var/lib/kubelet/pods/1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec/volumes" Aug 05 17:22:32.770765 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:32.770717 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q"] Aug 05 17:22:32.866324 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:32.866284 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:33.754081 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:33.754031 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:22:47.790921 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:47.790880 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" podUID="6f4a8a4e-2717-4199-a8c2-9da3171e546b" containerName="main" containerID="cri-o://763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5" gracePeriod=45 Aug 05 17:22:48.045766 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.045688 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:22:48.049443 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.049415 2575 generic.go:356] "Generic (PLEG): container finished" podID="6f4a8a4e-2717-4199-a8c2-9da3171e546b" containerID="763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5" exitCode=0 Aug 05 17:22:48.049614 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.049472 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" Aug 05 17:22:48.049614 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.049496 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" event={"ID":"6f4a8a4e-2717-4199-a8c2-9da3171e546b","Type":"ContainerDied","Data":"763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5"} Aug 05 17:22:48.049614 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.049558 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q" event={"ID":"6f4a8a4e-2717-4199-a8c2-9da3171e546b","Type":"ContainerDied","Data":"2cd62b1ffdfb78b28b42b3b85546487bce5d6598bfb4f0f3890096343de29114"} Aug 05 17:22:48.049614 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.049583 2575 scope.go:117] "RemoveContainer" containerID="763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5" Aug 05 17:22:48.061669 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.061641 2575 scope.go:117] "RemoveContainer" containerID="763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5" Aug 05 17:22:48.062203 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:22:48.062170 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5\": container with ID starting with 763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5 not found: ID does not exist" containerID="763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5" Aug 05 17:22:48.062324 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.062210 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5"} err="failed to get container status \"763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5\": rpc error: code = NotFound desc = could not find container \"763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5\": container with ID starting with 763fa7f9ba294c07dc2da3ff26b25e42e3e3f78fd2ca67a0c7d5cd543a814ad5 not found: ID does not exist" Aug 05 17:22:48.146294 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146263 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tmp-dir\") pod \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " Aug 05 17:22:48.146476 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146315 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-dshm\") pod \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " Aug 05 17:22:48.146476 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146350 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8864\" (UniqueName: \"kubernetes.io/projected/6f4a8a4e-2717-4199-a8c2-9da3171e546b-kube-api-access-c8864\") pod \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " Aug 05 17:22:48.146476 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146382 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-model-cache\") pod \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " Aug 05 17:22:48.146476 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146401 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-home\") pod \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " Aug 05 17:22:48.146476 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146431 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tls-certs\") pod \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\" (UID: \"6f4a8a4e-2717-4199-a8c2-9da3171e546b\") " Aug 05 17:22:48.146736 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146573 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f4a8a4e-2717-4199-a8c2-9da3171e546b" (UID: "6f4a8a4e-2717-4199-a8c2-9da3171e546b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:48.146736 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146605 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f4a8a4e-2717-4199-a8c2-9da3171e546b" (UID: "6f4a8a4e-2717-4199-a8c2-9da3171e546b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:48.146736 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146707 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-home" (OuterVolumeSpecName: "home") pod "6f4a8a4e-2717-4199-a8c2-9da3171e546b" (UID: "6f4a8a4e-2717-4199-a8c2-9da3171e546b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:48.146913 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146896 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-model-cache\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:22:48.146960 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146914 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-home\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:22:48.146960 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.146927 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tmp-dir\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:22:48.148807 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.148767 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-dshm" (OuterVolumeSpecName: "dshm") pod "6f4a8a4e-2717-4199-a8c2-9da3171e546b" (UID: "6f4a8a4e-2717-4199-a8c2-9da3171e546b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:48.148941 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.148827 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4a8a4e-2717-4199-a8c2-9da3171e546b-kube-api-access-c8864" (OuterVolumeSpecName: "kube-api-access-c8864") pod "6f4a8a4e-2717-4199-a8c2-9da3171e546b" (UID: "6f4a8a4e-2717-4199-a8c2-9da3171e546b"). InnerVolumeSpecName "kube-api-access-c8864". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:48.148941 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.148930 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f4a8a4e-2717-4199-a8c2-9da3171e546b" (UID: "6f4a8a4e-2717-4199-a8c2-9da3171e546b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:48.247780 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.247740 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4a8a4e-2717-4199-a8c2-9da3171e546b-dshm\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:22:48.247780 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.247775 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8864\" (UniqueName: \"kubernetes.io/projected/6f4a8a4e-2717-4199-a8c2-9da3171e546b-kube-api-access-c8864\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:22:48.248008 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.247785 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4a8a4e-2717-4199-a8c2-9da3171e546b-tls-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:22:48.373748 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.373701 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q"] Aug 05 17:22:48.378463 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:48.378434 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0b8087/ipp-route-0b8087-kserve-768bd4444c-pqf8q"] Aug 05 17:22:49.120375 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:49.120342 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4a8a4e-2717-4199-a8c2-9da3171e546b" path="/var/lib/kubelet/pods/6f4a8a4e-2717-4199-a8c2-9da3171e546b/volumes" Aug 05 17:22:57.366350 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:57.366315 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 17:22:57.373272 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:22:57.373249 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7c4d_c4f66f40-5075-4bbd-9f6d-95c6f63f00ec/ovn-acl-logging/0.log" Aug 05 17:23:09.568631 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:09.568596 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:10.874668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:10.874634 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:15.651590 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:15.651550 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:15.715576 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:15.715527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:15.940673 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:15.940575 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:16.187437 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.187394 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:16.311315 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.311271 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67"] Aug 05 17:23:16.311687 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.311673 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4a8a4e-2717-4199-a8c2-9da3171e546b" containerName="main" Aug 05 17:23:16.311732 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.311688 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4a8a4e-2717-4199-a8c2-9da3171e546b" containerName="main" Aug 05 17:23:16.311732 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.311707 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec" containerName="revoke-keys" Aug 05 17:23:16.311732 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.311714 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec" containerName="revoke-keys" Aug 05 17:23:16.311857 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.311771 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4a8a4e-2717-4199-a8c2-9da3171e546b" containerName="main" Aug 05 17:23:16.311857 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.311780 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b69cf03-7d4c-43cf-80e1-3d7f7a9466ec" containerName="revoke-keys" Aug 05 17:23:16.314700 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.314680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" Aug 05 17:23:16.317474 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.317449 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68wxh\"" Aug 05 17:23:16.324162 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.324136 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67"] Aug 05 17:23:16.437291 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.437239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edb7101-f61b-41e5-835a-d7c8b14afb7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67\" (UID: \"1edb7101-f61b-41e5-835a-d7c8b14afb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" Aug 05 17:23:16.538501 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.538452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edb7101-f61b-41e5-835a-d7c8b14afb7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67\" (UID: \"1edb7101-f61b-41e5-835a-d7c8b14afb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" Aug 05 17:23:16.539135 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.539113 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edb7101-f61b-41e5-835a-d7c8b14afb7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67\" (UID: \"1edb7101-f61b-41e5-835a-d7c8b14afb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" Aug 05 17:23:16.626486 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.626394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" Aug 05 17:23:16.767867 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:16.767839 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67"] Aug 05 17:23:17.204368 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:17.204331 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" event={"ID":"1edb7101-f61b-41e5-835a-d7c8b14afb7f","Type":"ContainerStarted","Data":"a8a89c6512117f435536889b7b3ad4eb16a566402e8cdfc106358716634ddcc4"} Aug 05 17:23:17.204559 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:17.204396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" event={"ID":"1edb7101-f61b-41e5-835a-d7c8b14afb7f","Type":"ContainerStarted","Data":"ab0606e4c4c5a3bbef542006280d170a31343d4759d0533397ee86d7d3b6961a"} Aug 05 17:23:17.228119 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:17.228060 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" podStartSLOduration=1.2280452 podStartE2EDuration="1.2280452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:17.225725339 +0000 UTC m=+3620.584977183" watchObservedRunningTime="2026-08-05 17:23:17.2280452 +0000 UTC m=+3620.587297043" Aug 05 17:23:18.211305 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:18.211269 2575 generic.go:356] "Generic (PLEG): container finished" podID="1edb7101-f61b-41e5-835a-d7c8b14afb7f" containerID="a8a89c6512117f435536889b7b3ad4eb16a566402e8cdfc106358716634ddcc4" exitCode=0 Aug 05 17:23:18.211700 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:18.211308 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" event={"ID":"1edb7101-f61b-41e5-835a-d7c8b14afb7f","Type":"ContainerDied","Data":"a8a89c6512117f435536889b7b3ad4eb16a566402e8cdfc106358716634ddcc4"} Aug 05 17:23:19.357783 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:19.357756 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" Aug 05 17:23:19.469284 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:19.469190 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edb7101-f61b-41e5-835a-d7c8b14afb7f-maas-api-service-ca\") pod \"1edb7101-f61b-41e5-835a-d7c8b14afb7f\" (UID: \"1edb7101-f61b-41e5-835a-d7c8b14afb7f\") " Aug 05 17:23:19.469512 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:19.469491 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edb7101-f61b-41e5-835a-d7c8b14afb7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1edb7101-f61b-41e5-835a-d7c8b14afb7f" (UID: "1edb7101-f61b-41e5-835a-d7c8b14afb7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:23:19.570273 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:19.570240 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edb7101-f61b-41e5-835a-d7c8b14afb7f-maas-api-service-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:20.223483 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:20.223456 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" Aug 05 17:23:20.223660 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:20.223456 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67" event={"ID":"1edb7101-f61b-41e5-835a-d7c8b14afb7f","Type":"ContainerDied","Data":"ab0606e4c4c5a3bbef542006280d170a31343d4759d0533397ee86d7d3b6961a"} Aug 05 17:23:20.223660 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:20.223581 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab0606e4c4c5a3bbef542006280d170a31343d4759d0533397ee86d7d3b6961a" Aug 05 17:23:27.015246 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.015212 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8"] Aug 05 17:23:27.015850 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.015475 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" podUID="6b10665a-0260-4acc-aa23-cc2567131800" containerName="maas-api" containerID="cri-o://815dd3a269e702e36b730fab0769fdc7e2c0f6f273cad44978f78501a4bddf3a" gracePeriod=30 Aug 05 17:23:27.078110 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.078073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:27.178989 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.178948 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:27.261145 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.261117 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67"] Aug 05 17:23:27.264559 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.264524 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:27.264755 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.264625 2575 generic.go:356] "Generic (PLEG): container finished" podID="6b10665a-0260-4acc-aa23-cc2567131800" containerID="815dd3a269e702e36b730fab0769fdc7e2c0f6f273cad44978f78501a4bddf3a" exitCode=0 Aug 05 17:23:27.264755 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.264711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" event={"ID":"6b10665a-0260-4acc-aa23-cc2567131800","Type":"ContainerDied","Data":"815dd3a269e702e36b730fab0769fdc7e2c0f6f273cad44978f78501a4bddf3a"} Aug 05 17:23:27.264937 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.264840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" event={"ID":"6b10665a-0260-4acc-aa23-cc2567131800","Type":"ContainerDied","Data":"4c7cc9885818928ef811a9a54122c69755afc912d3d48dfa77a9eb01e620f89c"} Aug 05 17:23:27.264937 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.264876 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c7cc9885818928ef811a9a54122c69755afc912d3d48dfa77a9eb01e620f89c" Aug 05 17:23:27.265312 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.265266 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ca1c80-wmk67"] Aug 05 17:23:27.273396 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.273368 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:23:27.331387 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.331352 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:27.343271 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.343239 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t56ns\" (UniqueName: \"kubernetes.io/projected/6b10665a-0260-4acc-aa23-cc2567131800-kube-api-access-t56ns\") pod \"6b10665a-0260-4acc-aa23-cc2567131800\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " Aug 05 17:23:27.343461 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.343281 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b10665a-0260-4acc-aa23-cc2567131800-openshift-service-ca\") pod \"6b10665a-0260-4acc-aa23-cc2567131800\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " Aug 05 17:23:27.343461 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.343301 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10665a-0260-4acc-aa23-cc2567131800-maas-api-tls\") pod \"6b10665a-0260-4acc-aa23-cc2567131800\" (UID: \"6b10665a-0260-4acc-aa23-cc2567131800\") " Aug 05 17:23:27.343701 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.343673 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b10665a-0260-4acc-aa23-cc2567131800-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b10665a-0260-4acc-aa23-cc2567131800" (UID: "6b10665a-0260-4acc-aa23-cc2567131800"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:23:27.345713 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.345677 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b10665a-0260-4acc-aa23-cc2567131800-kube-api-access-t56ns" (OuterVolumeSpecName: "kube-api-access-t56ns") pod "6b10665a-0260-4acc-aa23-cc2567131800" (UID: "6b10665a-0260-4acc-aa23-cc2567131800"). InnerVolumeSpecName "kube-api-access-t56ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:23:27.345870 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.345741 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b10665a-0260-4acc-aa23-cc2567131800-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b10665a-0260-4acc-aa23-cc2567131800" (UID: "6b10665a-0260-4acc-aa23-cc2567131800"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:23:27.443993 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.443960 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56ns\" (UniqueName: \"kubernetes.io/projected/6b10665a-0260-4acc-aa23-cc2567131800-kube-api-access-t56ns\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:27.443993 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.443991 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b10665a-0260-4acc-aa23-cc2567131800-openshift-service-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:27.443993 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.444000 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b10665a-0260-4acc-aa23-cc2567131800-maas-api-tls\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:27.642255 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:27.642159 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:28.271457 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:28.271426 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8" Aug 05 17:23:28.299804 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:28.299756 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8"] Aug 05 17:23:28.308427 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:28.308393 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca1c80-b45cd6898-tf2k8"] Aug 05 17:23:29.121228 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:29.121191 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1edb7101-f61b-41e5-835a-d7c8b14afb7f" path="/var/lib/kubelet/pods/1edb7101-f61b-41e5-835a-d7c8b14afb7f/volumes" Aug 05 17:23:29.121700 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:29.121677 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b10665a-0260-4acc-aa23-cc2567131800" path="/var/lib/kubelet/pods/6b10665a-0260-4acc-aa23-cc2567131800/volumes" Aug 05 17:23:29.344552 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:29.344514 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:23:33.399390 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:33.399351 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:33.474986 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:33.474945 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:33.659301 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:33.659204 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk"] Aug 05 17:23:33.659467 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:33.659425 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" containerID="cri-o://b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f" gracePeriod=30 Aug 05 17:23:33.686777 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:33.686742 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:33.899651 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:33.899616 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:38.912959 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.912930 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:23:38.947634 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.947551 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-envoy\") pod \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " Aug 05 17:23:38.947634 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.947595 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-socket\") pod \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " Aug 05 17:23:38.947634 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.947624 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istiod-ca-cert\") pod \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " Aug 05 17:23:38.947932 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.947650 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr6h6\" (UniqueName: \"kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-kube-api-access-cr6h6\") pod \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " Aug 05 17:23:38.947932 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.947696 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-credential-socket\") pod \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " Aug 05 17:23:38.947932 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.947729 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-data\") pod \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " Aug 05 17:23:38.947932 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.947765 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-token\") pod \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " Aug 05 17:23:38.947932 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.947813 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-certs\") pod \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " Aug 05 17:23:38.947932 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.947839 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-podinfo\") pod \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\" (UID: \"e9ed34e7-7386-442b-89ed-45e8ac707fd1\") " Aug 05 17:23:38.948228 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.948099 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9ed34e7-7386-442b-89ed-45e8ac707fd1" (UID: "e9ed34e7-7386-442b-89ed-45e8ac707fd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:23:38.948228 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.948172 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-credential-socket\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:38.948332 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.948232 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9ed34e7-7386-442b-89ed-45e8ac707fd1" (UID: "e9ed34e7-7386-442b-89ed-45e8ac707fd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:23:38.948389 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.948370 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9ed34e7-7386-442b-89ed-45e8ac707fd1" (UID: "e9ed34e7-7386-442b-89ed-45e8ac707fd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:23:38.948439 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.948420 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9ed34e7-7386-442b-89ed-45e8ac707fd1" (UID: "e9ed34e7-7386-442b-89ed-45e8ac707fd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:23:38.948487 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.948445 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9ed34e7-7386-442b-89ed-45e8ac707fd1" (UID: "e9ed34e7-7386-442b-89ed-45e8ac707fd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:23:38.950301 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.950267 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-kube-api-access-cr6h6" (OuterVolumeSpecName: "kube-api-access-cr6h6") pod "e9ed34e7-7386-442b-89ed-45e8ac707fd1" (UID: "e9ed34e7-7386-442b-89ed-45e8ac707fd1"). InnerVolumeSpecName "kube-api-access-cr6h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:23:38.950301 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.950266 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9ed34e7-7386-442b-89ed-45e8ac707fd1" (UID: "e9ed34e7-7386-442b-89ed-45e8ac707fd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:23:38.950469 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.950447 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9ed34e7-7386-442b-89ed-45e8ac707fd1" (UID: "e9ed34e7-7386-442b-89ed-45e8ac707fd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:23:38.950565 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:38.950548 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9ed34e7-7386-442b-89ed-45e8ac707fd1" (UID: "e9ed34e7-7386-442b-89ed-45e8ac707fd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:23:39.048645 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.048607 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-socket\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:39.048645 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.048640 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istiod-ca-cert\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:39.048645 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.048649 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr6h6\" (UniqueName: \"kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-kube-api-access-cr6h6\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:39.048953 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.048659 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-data\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:39.048953 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.048668 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-token\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:39.048953 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.048675 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-workload-certs\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:39.048953 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.048682 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-podinfo\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:39.048953 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.048689 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ed34e7-7386-442b-89ed-45e8ac707fd1-istio-envoy\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:39.329487 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.329451 2575 generic.go:356] "Generic (PLEG): container finished" podID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerID="b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f" exitCode=0 Aug 05 17:23:39.329666 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.329504 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" Aug 05 17:23:39.329666 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.329527 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" event={"ID":"e9ed34e7-7386-442b-89ed-45e8ac707fd1","Type":"ContainerDied","Data":"b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f"} Aug 05 17:23:39.329666 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.329585 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk" event={"ID":"e9ed34e7-7386-442b-89ed-45e8ac707fd1","Type":"ContainerDied","Data":"2a67ef990d30458b8fb873295eb533672cc6f6d6ef48a25ef4f3149ca281ef53"} Aug 05 17:23:39.329666 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.329608 2575 scope.go:117] "RemoveContainer" containerID="b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f" Aug 05 17:23:39.338894 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.338877 2575 scope.go:117] "RemoveContainer" containerID="cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d" Aug 05 17:23:39.347644 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.347619 2575 scope.go:117] "RemoveContainer" containerID="b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f" Aug 05 17:23:39.348317 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:23:39.348277 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f\": container with ID starting with b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f not found: ID does not exist" containerID="b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f" Aug 05 17:23:39.348440 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.348322 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f"} err="failed to get container status \"b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f\": rpc error: code = NotFound desc = could not find container \"b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f\": container with ID starting with b73d604648da5f0d30e5157af1d6456bd3836a126ce6c3d03744d4e05b804c9f not found: ID does not exist" Aug 05 17:23:39.348440 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.348347 2575 scope.go:117] "RemoveContainer" containerID="cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d" Aug 05 17:23:39.348676 ip-10-0-128-17 kubenswrapper[2575]: E0805 17:23:39.348656 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d\": container with ID starting with cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d not found: ID does not exist" containerID="cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d" Aug 05 17:23:39.348732 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.348692 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d"} err="failed to get container status \"cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d\": rpc error: code = NotFound desc = could not find container \"cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d\": container with ID starting with cf45cdbdc06d8105a478d521b84f0c9e2767153c752c75cb7fb9fc55d12d921d not found: ID does not exist" Aug 05 17:23:39.350174 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.350157 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk"] Aug 05 17:23:39.355856 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:39.355831 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca1c80-openshift-default-6b88fb9576-pg4xk"] Aug 05 17:23:41.120685 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:41.120648 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" path="/var/lib/kubelet/pods/e9ed34e7-7386-442b-89ed-45e8ac707fd1/volumes" Aug 05 17:23:44.661960 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:44.661921 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:44.745279 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:44.745243 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:44.844815 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:44.844762 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:44.913436 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:44.913341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:45.247319 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:45.247285 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:46.763118 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:46.763078 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:23:51.302404 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.302344 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:51.408029 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.407985 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:51.591355 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.591274 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:51.783627 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.783596 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:51.894363 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894284 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl"] Aug 05 17:23:51.894656 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894643 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b10665a-0260-4acc-aa23-cc2567131800" containerName="maas-api" Aug 05 17:23:51.894700 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894658 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b10665a-0260-4acc-aa23-cc2567131800" containerName="maas-api" Aug 05 17:23:51.894700 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894673 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" Aug 05 17:23:51.894700 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894680 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" Aug 05 17:23:51.894700 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894691 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1edb7101-f61b-41e5-835a-d7c8b14afb7f" containerName="revoke-keys" Aug 05 17:23:51.894700 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894697 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1edb7101-f61b-41e5-835a-d7c8b14afb7f" containerName="revoke-keys" Aug 05 17:23:51.894877 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894711 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" Aug 05 17:23:51.894877 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894717 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" Aug 05 17:23:51.894877 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894771 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" Aug 05 17:23:51.894877 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894778 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ed34e7-7386-442b-89ed-45e8ac707fd1" containerName="istio-proxy" Aug 05 17:23:51.894877 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894783 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1edb7101-f61b-41e5-835a-d7c8b14afb7f" containerName="revoke-keys" Aug 05 17:23:51.894877 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.894789 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b10665a-0260-4acc-aa23-cc2567131800" containerName="maas-api" Aug 05 17:23:51.898033 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.898016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl" Aug 05 17:23:51.901089 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.901071 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68wxh\"" Aug 05 17:23:51.901089 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.901078 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 17:23:51.905893 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.905823 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl"] Aug 05 17:23:51.968531 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:51.968487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902b78d7-f37c-4b93-9073-e11507e1f8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-57997e-c99dl\" (UID: \"902b78d7-f37c-4b93-9073-e11507e1f8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl" Aug 05 17:23:52.069355 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:52.069315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902b78d7-f37c-4b93-9073-e11507e1f8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-57997e-c99dl\" (UID: \"902b78d7-f37c-4b93-9073-e11507e1f8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl" Aug 05 17:23:52.070049 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:52.070023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902b78d7-f37c-4b93-9073-e11507e1f8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-57997e-c99dl\" (UID: \"902b78d7-f37c-4b93-9073-e11507e1f8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl" Aug 05 17:23:52.209721 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:52.209628 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl" Aug 05 17:23:52.351985 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:52.351901 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl"] Aug 05 17:23:52.357753 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:23:52.357728 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902b78d7_f37c_4b93_9073_e11507e1f8cb.slice/crio-b956e6128598ab392056253f42bd86d7185db01ad1e54dca52370628ecaf6f8f WatchSource:0}: Error finding container b956e6128598ab392056253f42bd86d7185db01ad1e54dca52370628ecaf6f8f: Status 404 returned error can't find the container with id b956e6128598ab392056253f42bd86d7185db01ad1e54dca52370628ecaf6f8f Aug 05 17:23:52.398876 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:52.398828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl" event={"ID":"902b78d7-f37c-4b93-9073-e11507e1f8cb","Type":"ContainerStarted","Data":"b956e6128598ab392056253f42bd86d7185db01ad1e54dca52370628ecaf6f8f"} Aug 05 17:23:53.405974 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:53.405882 2575 generic.go:356] "Generic (PLEG): container finished" podID="902b78d7-f37c-4b93-9073-e11507e1f8cb" containerID="ceceb846dbdf7aaf04c2adffcf07592eca7ffb80b48846749172fc881c10cfbd" exitCode=0 Aug 05 17:23:53.405974 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:53.405954 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl" event={"ID":"902b78d7-f37c-4b93-9073-e11507e1f8cb","Type":"ContainerDied","Data":"ceceb846dbdf7aaf04c2adffcf07592eca7ffb80b48846749172fc881c10cfbd"} Aug 05 17:23:54.544613 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:54.544585 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl" Aug 05 17:23:54.695595 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:54.695502 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902b78d7-f37c-4b93-9073-e11507e1f8cb-maas-api-service-ca\") pod \"902b78d7-f37c-4b93-9073-e11507e1f8cb\" (UID: \"902b78d7-f37c-4b93-9073-e11507e1f8cb\") " Aug 05 17:23:54.695913 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:54.695887 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902b78d7-f37c-4b93-9073-e11507e1f8cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "902b78d7-f37c-4b93-9073-e11507e1f8cb" (UID: "902b78d7-f37c-4b93-9073-e11507e1f8cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:23:54.796678 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:54.796634 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902b78d7-f37c-4b93-9073-e11507e1f8cb-maas-api-service-ca\") on node \"ip-10-0-128-17.ec2.internal\" DevicePath \"\"" Aug 05 17:23:55.417122 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:55.417041 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl" Aug 05 17:23:55.417122 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:55.417092 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl" event={"ID":"902b78d7-f37c-4b93-9073-e11507e1f8cb","Type":"ContainerDied","Data":"b956e6128598ab392056253f42bd86d7185db01ad1e54dca52370628ecaf6f8f"} Aug 05 17:23:55.417302 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:23:55.417135 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b956e6128598ab392056253f42bd86d7185db01ad1e54dca52370628ecaf6f8f" Aug 05 17:24:02.489438 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:02.489387 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:02.641502 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:02.641453 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:02.735454 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:02.735416 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:02.754287 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:02.754189 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl"] Aug 05 17:24:02.756240 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:02.756216 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-57997e-c99dl"] Aug 05 17:24:02.799709 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:02.799674 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:03.120370 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:03.120289 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902b78d7-f37c-4b93-9073-e11507e1f8cb" path="/var/lib/kubelet/pods/902b78d7-f37c-4b93-9073-e11507e1f8cb/volumes" Aug 05 17:24:03.123575 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:03.123543 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:04.748967 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:04.748920 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-zf2hr"] Aug 05 17:24:09.329368 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:09.329329 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:09.429235 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:09.429195 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:09.609558 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:09.609456 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:09.806497 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:09.806458 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:10.210959 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:10.210925 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bb787fc85-5hbbh_8dfe9c65-c1c9-42c6-9be0-5a25cbb26430/authorino/0.log" Aug 05 17:24:16.105303 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:16.105270 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfc779b6-f6hsj_8b5c41a9-f9d5-4844-8580-26c0f3f53edc/copy-manifests/0.log" Aug 05 17:24:16.116981 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:16.116951 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfc779b6-f6hsj_8b5c41a9-f9d5-4844-8580-26c0f3f53edc/manager/0.log" Aug 05 17:24:17.907775 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:17.907695 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bb787fc85-5hbbh_8dfe9c65-c1c9-42c6-9be0-5a25cbb26430/authorino/0.log" Aug 05 17:24:18.476431 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:18.476399 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zf2hr_d8b8fa28-e50c-4a67-9f58-13df531c13c7/limitador/0.log" Aug 05 17:24:18.965065 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:18.965032 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f_15812cb0-a4b0-45c3-aeb1-7c73223b99c9/istio-proxy/0.log" Aug 05 17:24:19.316941 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:19.316905 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65879f568d-tpmh6_098f0277-617a-4761-bfed-e583eab271ff/kube-auth-proxy/0.log" Aug 05 17:24:21.668023 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:21.667976 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-wpx45_30d24b56-d8cf-4c23-ac86-47a8e15be529/main/0.log" Aug 05 17:24:26.502668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:26.502635 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:26.602321 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:26.602280 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:29.318783 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:29.318745 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c5l2j_8ddce42c-0420-4820-9d02-f1bfbdd3bf42/global-pull-secret-syncer/0.log" Aug 05 17:24:29.469773 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:29.469732 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" Aug 05 17:24:29.564467 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:29.564435 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-17.ec2.internal_b405ac74232cf257989af7a5ac3a33f2/haproxy/0.log" Aug 05 17:24:34.939250 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:34.939211 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bb787fc85-5hbbh_8dfe9c65-c1c9-42c6-9be0-5a25cbb26430/authorino/0.log" Aug 05 17:24:35.215277 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:35.215198 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-zf2hr_d8b8fa28-e50c-4a67-9f58-13df531c13c7/limitador/0.log" Aug 05 17:24:36.851055 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:36.851000 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ef73249-a007-4cfb-b75c-a5c4e65ff9b4/alertmanager/0.log" Aug 05 17:24:36.909567 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:36.909527 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ef73249-a007-4cfb-b75c-a5c4e65ff9b4/config-reloader/0.log" Aug 05 17:24:36.947004 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:36.946975 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ef73249-a007-4cfb-b75c-a5c4e65ff9b4/kube-rbac-proxy-web/0.log" Aug 05 17:24:36.975222 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:36.975130 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ef73249-a007-4cfb-b75c-a5c4e65ff9b4/kube-rbac-proxy/0.log" Aug 05 17:24:36.999168 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:36.999138 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ef73249-a007-4cfb-b75c-a5c4e65ff9b4/kube-rbac-proxy-metric/0.log" Aug 05 17:24:37.022842 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.022809 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ef73249-a007-4cfb-b75c-a5c4e65ff9b4/prom-label-proxy/0.log" Aug 05 17:24:37.052368 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.052342 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ef73249-a007-4cfb-b75c-a5c4e65ff9b4/init-config-reloader/0.log" Aug 05 17:24:37.131438 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.131398 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-v4wtw_57aad5a8-d382-423d-940e-b36b93ec0ef4/kube-state-metrics/0.log" Aug 05 17:24:37.149084 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.149047 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-v4wtw_57aad5a8-d382-423d-940e-b36b93ec0ef4/kube-rbac-proxy-main/0.log" Aug 05 17:24:37.170081 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.170036 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-v4wtw_57aad5a8-d382-423d-940e-b36b93ec0ef4/kube-rbac-proxy-self/0.log" Aug 05 17:24:37.271406 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.271367 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6qc2p_22184a85-e17b-49a9-b102-58bf08245776/node-exporter/0.log" Aug 05 17:24:37.296668 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.296637 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6qc2p_22184a85-e17b-49a9-b102-58bf08245776/kube-rbac-proxy/0.log" Aug 05 17:24:37.321852 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.321830 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6qc2p_22184a85-e17b-49a9-b102-58bf08245776/init-textfile/0.log" Aug 05 17:24:37.831736 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.831702 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-gqnwh_8575379f-09e6-4855-ae41-e804a5ee8a2f/prometheus-operator/0.log" Aug 05 17:24:37.852196 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.852169 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-gqnwh_8575379f-09e6-4855-ae41-e804a5ee8a2f/kube-rbac-proxy/0.log" Aug 05 17:24:37.883500 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.883469 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-v4dhf_29d89fc0-da14-4af8-9d43-affbb48319eb/prometheus-operator-admission-webhook/0.log" Aug 05 17:24:37.917635 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.917598 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b7fc9bb58-s7gcl_65f07a07-c83c-4433-bf9a-64dd7cd5f2a6/telemeter-client/0.log" Aug 05 17:24:37.939532 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.939497 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b7fc9bb58-s7gcl_65f07a07-c83c-4433-bf9a-64dd7cd5f2a6/reload/0.log" Aug 05 17:24:37.965690 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:37.965661 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b7fc9bb58-s7gcl_65f07a07-c83c-4433-bf9a-64dd7cd5f2a6/kube-rbac-proxy/0.log" Aug 05 17:24:38.001395 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:38.001366 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-745bfdcf87-f7fxw_9f02957a-9c2d-4cce-a040-eb29b0fbf790/thanos-query/0.log" Aug 05 17:24:38.024244 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:38.024216 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-745bfdcf87-f7fxw_9f02957a-9c2d-4cce-a040-eb29b0fbf790/kube-rbac-proxy-web/0.log" Aug 05 17:24:38.047151 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:38.047124 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-745bfdcf87-f7fxw_9f02957a-9c2d-4cce-a040-eb29b0fbf790/kube-rbac-proxy/0.log" Aug 05 17:24:38.067549 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:38.067523 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-745bfdcf87-f7fxw_9f02957a-9c2d-4cce-a040-eb29b0fbf790/prom-label-proxy/0.log" Aug 05 17:24:38.088760 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:38.088680 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-745bfdcf87-f7fxw_9f02957a-9c2d-4cce-a040-eb29b0fbf790/kube-rbac-proxy-rules/0.log" Aug 05 17:24:38.110197 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:38.110171 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-745bfdcf87-f7fxw_9f02957a-9c2d-4cce-a040-eb29b0fbf790/kube-rbac-proxy-metrics/0.log" Aug 05 17:24:39.364710 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.364679 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-zqvmc_07165f9c-e008-4e15-bf44-24e63eb41f3b/networking-console-plugin/0.log" Aug 05 17:24:39.525269 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.525232 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j"] Aug 05 17:24:39.525670 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.525651 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902b78d7-f37c-4b93-9073-e11507e1f8cb" containerName="revoke-keys" Aug 05 17:24:39.525752 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.525673 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="902b78d7-f37c-4b93-9073-e11507e1f8cb" containerName="revoke-keys" Aug 05 17:24:39.525836 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.525821 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="902b78d7-f37c-4b93-9073-e11507e1f8cb" containerName="revoke-keys" Aug 05 17:24:39.529189 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.529165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.532019 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.531999 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pbkcv\"/\"openshift-service-ca.crt\"" Aug 05 17:24:39.533293 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.533276 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pbkcv\"/\"kube-root-ca.crt\"" Aug 05 17:24:39.533406 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.533311 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pbkcv\"/\"default-dockercfg-fv2mn\"" Aug 05 17:24:39.538912 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.538880 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j"] Aug 05 17:24:39.602226 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.602191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tlms\" (UniqueName: \"kubernetes.io/projected/3cae2292-fa23-4acf-b253-6865472165dd-kube-api-access-9tlms\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.602391 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.602245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-proc\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.602391 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.602313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-sys\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.602391 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.602350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-lib-modules\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.602391 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.602380 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-podres\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.702960 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.702867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tlms\" (UniqueName: \"kubernetes.io/projected/3cae2292-fa23-4acf-b253-6865472165dd-kube-api-access-9tlms\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.702960 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.702943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-proc\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.703155 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.702978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-sys\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.703155 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.703004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-lib-modules\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.703155 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.703039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-podres\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.703155 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.703087 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-sys\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.703155 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.703086 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-proc\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.703321 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.703176 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-lib-modules\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.703321 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.703181 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cae2292-fa23-4acf-b253-6865472165dd-podres\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.712639 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.712605 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tlms\" (UniqueName: \"kubernetes.io/projected/3cae2292-fa23-4acf-b253-6865472165dd-kube-api-access-9tlms\") pod \"perf-node-gather-daemonset-66p7j\" (UID: \"3cae2292-fa23-4acf-b253-6865472165dd\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.840463 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.840422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:39.981060 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:39.981034 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j"] Aug 05 17:24:39.983513 ip-10-0-128-17 kubenswrapper[2575]: W0805 17:24:39.983483 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3cae2292_fa23_4acf_b253_6865472165dd.slice/crio-6c0dbd287e5ca76e237e9da3bdea8412229225cf025049925a0e0ce03ebd6c97 WatchSource:0}: Error finding container 6c0dbd287e5ca76e237e9da3bdea8412229225cf025049925a0e0ce03ebd6c97: Status 404 returned error can't find the container with id 6c0dbd287e5ca76e237e9da3bdea8412229225cf025049925a0e0ce03ebd6c97 Aug 05 17:24:40.620404 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:40.620372 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bdc47958d-vvkq5_913c732e-f371-48b6-8b00-caa7f8264b88/console/0.log" Aug 05 17:24:40.646282 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:40.646245 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" event={"ID":"3cae2292-fa23-4acf-b253-6865472165dd","Type":"ContainerStarted","Data":"c8a4981fb5c4f3840f2b751ec622f4a002e05c351069db2da439165dc6331f4f"} Aug 05 17:24:40.646483 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:40.646321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" event={"ID":"3cae2292-fa23-4acf-b253-6865472165dd","Type":"ContainerStarted","Data":"6c0dbd287e5ca76e237e9da3bdea8412229225cf025049925a0e0ce03ebd6c97"} Aug 05 17:24:40.646483 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:40.646347 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:40.667103 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:40.667026 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" podStartSLOduration=1.667007743 podStartE2EDuration="1.667007743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:40.665737923 +0000 UTC m=+3704.024989777" watchObservedRunningTime="2026-08-05 17:24:40.667007743 +0000 UTC m=+3704.026259586" Aug 05 17:24:40.667874 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:40.667844 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-wv2fb_7e9325ac-4eaa-42c5-8bd1-5e35a32d6629/download-server/0.log" Aug 05 17:24:40.850894 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:40.850856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-559j8_b60b5da1-b029-4644-b783-f7a393454913/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:42.430516 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:42.430488 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b7drl_049c9ec1-4759-426e-9290-5be491ba3f71/dns/0.log" Aug 05 17:24:42.453122 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:42.453091 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b7drl_049c9ec1-4759-426e-9290-5be491ba3f71/kube-rbac-proxy/0.log" Aug 05 17:24:42.577767 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:42.577710 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-22g2w_d3c6ce5d-ec80-437e-a440-8cc72d6de221/dns-node-resolver/0.log" Aug 05 17:24:43.223193 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:43.223166 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6ff4748cdb-cslpc_bc887723-951c-470f-99ce-7684edad114b/registry/0.log" Aug 05 17:24:43.312297 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:43.312268 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-stg9n_4873e7f5-62e1-4a7f-99c8-1f67a7fd86e3/node-ca/0.log" Aug 05 17:24:44.343382 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:44.343347 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f_15812cb0-a4b0-45c3-aeb1-7c73223b99c9/istio-proxy/0.log" Aug 05 17:24:44.561837 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:44.561783 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65879f568d-tpmh6_098f0277-617a-4761-bfed-e583eab271ff/kube-auth-proxy/0.log" Aug 05 17:24:45.625909 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:45.625876 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s4ndk_2234a147-bfa6-4060-90f2-83899121eadd/serve-healthcheck-canary/0.log" Aug 05 17:24:46.242161 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:46.242131 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d59b8_7f7aff24-8650-49c3-99a6-d2c7cce19769/kube-rbac-proxy/0.log" Aug 05 17:24:46.265566 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:46.265536 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d59b8_7f7aff24-8650-49c3-99a6-d2c7cce19769/exporter/0.log" Aug 05 17:24:46.288004 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:46.287969 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d59b8_7f7aff24-8650-49c3-99a6-d2c7cce19769/extractor/0.log" Aug 05 17:24:46.662699 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:46.662627 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-66p7j" Aug 05 17:24:49.392367 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:49.392336 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfc779b6-f6hsj_8b5c41a9-f9d5-4844-8580-26c0f3f53edc/manager/0.log" Aug 05 17:24:49.417563 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:49.417531 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfc779b6-f6hsj_8b5c41a9-f9d5-4844-8580-26c0f3f53edc/copy-manifests/0.log" Aug 05 17:24:50.329192 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:50.329160 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f_15812cb0-a4b0-45c3-aeb1-7c73223b99c9/istio-proxy/0.log" Aug 05 17:24:50.524089 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:50.524035 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65879f568d-tpmh6_098f0277-617a-4761-bfed-e583eab271ff/kube-auth-proxy/0.log" Aug 05 17:24:50.846070 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:50.846025 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bfcf8b475-cs66r_66510f0e-ec14-4184-a04d-a13767463593/manager/0.log" Aug 05 17:24:52.068207 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:52.068171 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6r58f_15812cb0-a4b0-45c3-aeb1-7c73223b99c9/istio-proxy/0.log" Aug 05 17:24:52.267862 ip-10-0-128-17 kubenswrapper[2575]: I0805 17:24:52.267830 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65879f568d-tpmh6_098f0277-617a-4761-bfed-e583eab271ff/kube-auth-proxy/0.log"