Jul 15 10:19:57.756022 ip-10-0-133-247 systemd[1]: Starting Kubernetes Kubelet... Jul 15 10:19:58.131109 ip-10-0-133-247 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 10:19:58.131109 ip-10-0-133-247 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 10:19:58.131109 ip-10-0-133-247 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 10:19:58.131109 ip-10-0-133-247 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 10:19:58.131109 ip-10-0-133-247 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 10:19:58.131642 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.131104 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 10:19:58.134410 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134395 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:19:58.134410 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134409 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134413 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134418 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134421 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134425 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134427 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134430 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134433 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134435 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134438 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134441 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134444 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134446 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134450 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134455 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134457 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134460 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134463 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134465 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:19:58.134474 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134468 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134470 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134474 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134478 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134482 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134485 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134488 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134491 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134494 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134496 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134499 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134502 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134505 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134507 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134510 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134513 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134515 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134518 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134520 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:19:58.134942 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134523 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134525 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134527 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134530 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134533 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134535 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134538 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134540 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134543 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134545 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134547 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134550 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134552 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134555 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134559 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134563 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134565 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134568 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134570 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134573 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:19:58.135388 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134576 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134578 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134581 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134583 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134586 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134589 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134591 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134593 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134596 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134599 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134601 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134604 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134606 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134609 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134611 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134613 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134616 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134620 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134623 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134625 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:19:58.135963 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134628 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134630 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134633 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134635 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134638 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134640 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134643 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134646 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134649 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134651 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134654 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134656 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134659 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134662 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134666 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134668 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134671 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134673 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134676 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:19:58.136441 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.134678 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135342 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135348 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135350 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135353 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135355 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135358 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135361 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135363 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135366 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135369 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135373 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135376 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135379 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135381 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135385 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135389 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135391 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135394 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:19:58.136913 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135397 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135400 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135402 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135405 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135408 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135410 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135413 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135416 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135418 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135420 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135423 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135426 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135428 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135430 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135433 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135435 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135438 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135441 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135443 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135446 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:19:58.137378 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135449 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135451 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135454 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135457 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135461 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135463 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135466 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135469 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135471 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135474 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135476 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135479 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135483 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135486 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135489 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135491 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135494 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135497 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135499 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:19:58.137876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135502 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135504 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135507 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135509 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135511 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135514 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135516 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135518 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135527 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135530 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135532 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135534 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135537 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135539 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135543 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135545 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135548 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135550 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135553 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135556 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:19:58.138334 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135558 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135560 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135563 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135566 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135568 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135572 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135576 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135579 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135582 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135585 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135587 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135590 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135593 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135595 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135598 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135600 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135603 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135605 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135607 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:19:58.138833 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135610 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135615 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135618 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135977 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135982 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135986 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135988 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135991 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135994 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135996 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.135999 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136002 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136004 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136007 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136010 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136012 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136015 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136017 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136020 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136022 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:19:58.139293 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136025 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136028 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136030 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136033 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136036 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136039 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136043 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136046 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136050 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136052 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136056 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136058 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136061 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136067 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136070 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136072 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136076 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136079 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:19:58.139771 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136082 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136085 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136088 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136090 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136093 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136096 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136099 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136102 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136104 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136107 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136110 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136112 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136115 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136118 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136121 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136123 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136126 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136129 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136131 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136134 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:19:58.140249 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136136 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136138 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136141 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136144 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136146 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136149 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136151 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136156 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136158 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136161 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136163 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136166 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136169 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136171 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136174 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136177 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136180 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136182 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136185 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136188 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:19:58.140720 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136191 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136194 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136196 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136199 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136202 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136204 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136207 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136209 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136212 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136214 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136217 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136219 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136221 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136224 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136226 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136229 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136231 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136234 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136236 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:19:58.141210 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136241 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136243 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136246 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136248 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136251 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136319 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136325 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136332 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136338 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136343 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136348 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136354 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136360 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136365 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136368 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136372 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136375 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136378 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136381 2579 flags.go:64] FLAG: --cgroup-root="" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136384 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136387 2579 flags.go:64] FLAG: --client-ca-file="" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136389 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136392 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 10:19:58.141682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136396 2579 flags.go:64] FLAG: --cluster-domain="" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136399 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136402 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136405 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136409 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136412 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136415 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136418 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136421 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136427 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136430 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136433 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136436 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136439 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136442 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136446 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136449 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136451 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136455 2579 flags.go:64] FLAG: --enable-server="true" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136458 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136462 2579 flags.go:64] FLAG: --event-burst="100" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136465 2579 flags.go:64] FLAG: --event-qps="50" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136468 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136471 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 10:19:58.142247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136474 2579 flags.go:64] FLAG: --eviction-hard="" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136478 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136481 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136484 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136487 2579 flags.go:64] FLAG: --eviction-soft="" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136490 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136492 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136495 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136498 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136500 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136503 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136506 2579 flags.go:64] FLAG: --feature-gates="" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136509 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136512 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136515 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136518 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136521 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136526 2579 flags.go:64] FLAG: --help="false" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136529 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136532 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136535 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136538 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136541 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 10:19:58.142853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136544 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136547 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136550 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136552 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136555 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136558 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136561 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136563 2579 flags.go:64] FLAG: --kube-reserved="" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136566 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136569 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136572 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136575 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136577 2579 flags.go:64] FLAG: --lock-file="" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136580 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136583 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136586 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136591 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136593 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136596 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136599 2579 flags.go:64] FLAG: --logging-format="text" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136601 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136604 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136607 2579 flags.go:64] FLAG: --manifest-url="" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136610 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136614 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 10:19:58.143429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136616 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136622 2579 flags.go:64] FLAG: --max-pods="110" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136625 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136628 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136631 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136633 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136636 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136639 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136642 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136653 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136656 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136659 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136663 2579 flags.go:64] FLAG: --pod-cidr="" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136665 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136670 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136673 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136676 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136679 2579 flags.go:64] FLAG: --port="10250" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136682 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136685 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05230210d325fd511" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136688 2579 flags.go:64] FLAG: --qos-reserved="" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136691 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136694 2579 flags.go:64] FLAG: --register-node="true" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136696 2579 flags.go:64] FLAG: --register-with-taints="" Jul 15 10:19:58.144099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136699 2579 flags.go:64] FLAG: --registry-burst="10" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136702 2579 flags.go:64] FLAG: --registry-qps="5" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136705 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136708 2579 flags.go:64] FLAG: --reserved-memory="" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136711 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136714 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136717 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136720 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136722 2579 flags.go:64] FLAG: --runonce="false" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136727 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136731 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136734 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136750 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136753 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136756 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136759 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136762 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136765 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136768 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136771 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136774 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136777 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136779 2579 flags.go:64] FLAG: --system-cgroups="" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136782 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136787 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 10:19:58.144679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136790 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136793 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136797 2579 flags.go:64] FLAG: --tls-min-version="" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136799 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136802 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136805 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136807 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136810 2579 flags.go:64] FLAG: --v="2" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136814 2579 flags.go:64] FLAG: --version="false" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136818 2579 flags.go:64] FLAG: --vmodule="" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136821 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.136824 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136968 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136972 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136975 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136978 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136982 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136985 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136987 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136990 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136993 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136995 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:19:58.145290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.136997 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137000 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137003 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137005 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137008 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137010 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137013 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137016 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137018 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137021 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137023 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137026 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137029 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137031 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137036 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137040 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137043 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137045 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137048 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:19:58.145825 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137051 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137053 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137056 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137059 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137061 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137065 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137069 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137072 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137076 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137079 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137082 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137085 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137087 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137090 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137094 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137097 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137100 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137103 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137105 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:19:58.146290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137108 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137111 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137114 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137116 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137119 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137121 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137124 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137127 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137130 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137132 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137135 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137137 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137140 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137143 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137145 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137148 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137150 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137153 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137155 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137157 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:19:58.146822 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137160 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137163 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137166 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137169 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137171 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137174 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137177 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137179 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137182 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137184 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137187 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137190 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137192 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137195 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137197 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137199 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137202 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137205 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137207 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137210 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:19:58.147304 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137213 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137215 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137218 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137220 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137223 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137225 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137228 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137230 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137233 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137235 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.137237 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.138184 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.145894 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.145909 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 10:19:58.147793 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.145927 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.145936 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.145999 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146004 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146008 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146011 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146014 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146017 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146020 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146023 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146025 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146028 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146030 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146033 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146035 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146038 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146040 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146043 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146045 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:19:58.148138 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146048 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146051 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146053 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146056 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146058 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146061 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146063 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146066 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146068 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146070 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146073 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146076 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146080 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146083 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146086 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146088 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146091 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146093 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146097 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:19:58.148582 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146101 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146104 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146107 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146110 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146113 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146116 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146118 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146121 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146123 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146126 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146128 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146131 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146133 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146135 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146138 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146140 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146143 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146145 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146148 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146150 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:19:58.149059 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146152 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146155 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146157 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146161 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146165 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146169 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146172 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146175 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146178 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146180 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146183 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146186 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146188 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146191 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146193 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146195 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146198 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146201 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146203 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:19:58.149527 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146207 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146210 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146212 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146215 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146217 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146220 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146222 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146224 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146227 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146230 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146232 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146235 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146237 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146240 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146242 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146244 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146247 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146250 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146252 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146255 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:19:58.150052 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146258 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146261 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146263 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146265 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.146270 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146398 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146403 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146406 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146409 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146412 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146415 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146418 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146421 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146423 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146426 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:19:58.150530 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146428 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146431 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146433 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146436 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146438 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146441 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146444 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146446 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146448 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146451 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146453 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146456 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146459 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146462 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146465 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146467 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146470 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146472 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146475 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146478 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:19:58.150914 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146480 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146482 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146485 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146487 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146489 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146492 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146495 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146497 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146500 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146502 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146505 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146507 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146510 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146512 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146514 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146517 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146519 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146522 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146524 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146527 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:19:58.151387 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146529 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146531 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146534 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146536 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146539 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146541 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146544 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146547 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146551 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146553 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146556 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146559 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146561 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146564 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146566 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146569 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146571 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146574 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146576 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146578 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:19:58.151906 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146581 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146583 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146586 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146588 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146591 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146593 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146597 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146600 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146603 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146607 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146609 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146612 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146614 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146617 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146619 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146622 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146624 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146627 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146629 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:19:58.152379 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146632 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146634 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146636 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146640 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146643 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146646 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146650 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146652 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146655 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:58.146658 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.146662 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.146782 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.151433 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 10:19:58.152983 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.152937 2579 server.go:1019] "Starting client certificate rotation" Jul 15 10:19:58.153304 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.153036 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 10:19:58.153304 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.153065 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 10:19:58.156311 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.156095 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 10:19:58.158046 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.158029 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 10:19:58.168867 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.168844 2579 log.go:25] "Validated CRI v1 runtime API" Jul 15 10:19:58.171133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.171120 2579 log.go:25] "Validated CRI v1 image API" Jul 15 10:19:58.172568 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.172552 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 10:19:58.175997 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.175980 2579 fs.go:135] Filesystem UUIDs: map[0af0a493-54ba-4cdb-9b45-c14200fd20f0:/dev/nvme0n1p3 50952afb-f790-40e3-a116-cc8c11dc920d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 15 10:19:58.176041 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.175997 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 10:19:58.180475 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.180458 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 10:19:58.180955 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.180853 2579 manager.go:216] Machine: {Timestamp:2026-07-15 10:19:58.179525978 +0000 UTC m=+0.312719319 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097713 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b1f0d46518a07aa943cb82fe4bbe2 SystemUUID:ec2b1f0d-4651-8a07-aa94-3cb82fe4bbe2 BootID:db43f809-d9e4-44fc-9421-445791bf9075 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:45:b4:af:fe:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:45:b4:af:fe:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:ec:7c:1d:05:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 10:19:58.180955 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.180950 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 10:19:58.181046 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.181018 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 10:19:58.183830 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.183808 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 10:19:58.183951 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.183832 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-247.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 10:19:58.185107 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.185097 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 10:19:58.185138 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.185110 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 10:19:58.185138 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.185123 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 10:19:58.185660 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.185650 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 10:19:58.186407 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.186398 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 10:19:58.189405 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.189393 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 15 10:19:58.189513 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.189503 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 10:19:58.189546 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.189535 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 15 10:19:58.189577 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.189550 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 10:19:58.189577 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.189563 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 10:19:58.189577 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.189572 2579 kubelet.go:398] "Adding apiserver pod source" Jul 15 10:19:58.189680 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.189579 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 10:19:58.189956 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.189945 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 10:19:58.190009 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.189964 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 10:19:58.194659 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.194641 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 10:19:58.196505 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.196488 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 10:19:58.196596 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.196526 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 10:19:58.197069 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197056 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 10:19:58.197069 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197077 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 10:19:58.197160 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197086 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 10:19:58.197160 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197094 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 10:19:58.197160 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197105 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 10:19:58.197160 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197113 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 10:19:58.197315 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197305 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 10:19:58.197365 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197318 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 10:19:58.197365 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197329 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 10:19:58.197365 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197337 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 10:19:58.197365 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197349 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 10:19:58.197365 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197361 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 10:19:58.197590 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197391 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 10:19:58.197590 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.197400 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 10:19:58.199585 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.199562 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-247.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 10:19:58.199666 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.199642 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-247.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 10:19:58.199721 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.199687 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 10:19:58.200462 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.200445 2579 server.go:1268] "Started kubelet" Jul 15 10:19:58.200569 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.200541 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 10:19:58.201153 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.201087 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 10:19:58.201239 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.201161 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 10:19:58.201797 ip-10-0-133-247 systemd[1]: Started Kubernetes Kubelet. Jul 15 10:19:58.203262 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.203245 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 15 10:19:58.205159 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.205140 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 10:19:58.205706 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.205687 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 10:19:58.210217 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.210201 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phpl5" Jul 15 10:19:58.210587 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.210568 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 10:19:58.210673 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.210572 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 10:19:58.210673 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.210667 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:58.211164 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211149 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 10:19:58.211268 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211151 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 10:19:58.211364 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211354 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 10:19:58.211563 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211546 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 15 10:19:58.211563 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211564 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 15 10:19:58.211659 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211618 2579 factory.go:153] Registering CRI-O factory Jul 15 10:19:58.211659 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211629 2579 factory.go:223] Registration of the crio container factory successfully Jul 15 10:19:58.211761 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211692 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 10:19:58.211761 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211701 2579 factory.go:55] Registering systemd factory Jul 15 10:19:58.211761 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211709 2579 factory.go:223] Registration of the systemd container factory successfully Jul 15 10:19:58.211761 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211728 2579 factory.go:103] Registering Raw factory Jul 15 10:19:58.211761 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.211762 2579 manager.go:1196] Started watching for new ooms in manager Jul 15 10:19:58.212050 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.212037 2579 manager.go:349] Starting recovery of all containers Jul 15 10:19:58.214518 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.214489 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-247.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 10:19:58.214645 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.214625 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 10:19:58.215475 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.214501 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-247.ec2.internal.18c26f517eacadc6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-247.ec2.internal,UID:ip-10-0-133-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-247.ec2.internal,},FirstTimestamp:2026-07-15 10:19:58.200417734 +0000 UTC m=+0.333611063,LastTimestamp:2026-07-15 10:19:58.200417734 +0000 UTC m=+0.333611063,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-247.ec2.internal,}" Jul 15 10:19:58.217710 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.217687 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phpl5" Jul 15 10:19:58.225773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.225620 2579 manager.go:354] Recovery completed Jul 15 10:19:58.230218 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.230206 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:19:58.232022 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.232006 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:19:58.232093 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.232054 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:19:58.232093 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.232064 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:19:58.232578 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.232564 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 10:19:58.232578 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.232572 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:19:58.232706 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.232575 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 10:19:58.232706 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.232655 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 15 10:19:58.233198 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.233184 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:19:58.233259 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.233209 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:19:58.233259 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.233221 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:19:58.233810 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.233798 2579 policy_none.go:49] "None policy: Start" Jul 15 10:19:58.233864 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.233815 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 10:19:58.233864 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.233829 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 10:19:58.234264 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.234208 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-247.ec2.internal.18c26f51808ee6da default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-247.ec2.internal,UID:ip-10-0-133-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-247.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-247.ec2.internal,},FirstTimestamp:2026-07-15 10:19:58.232020698 +0000 UTC m=+0.365214029,LastTimestamp:2026-07-15 10:19:58.232020698 +0000 UTC m=+0.365214029,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-247.ec2.internal,}" Jul 15 10:19:58.234995 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.234983 2579 policy_none.go:47] "Start" Jul 15 10:19:58.264277 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.264258 2579 manager.go:337] "Starting Device Plugin manager" Jul 15 10:19:58.264367 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.264290 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 10:19:58.264367 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.264301 2579 server.go:87] "Starting device plugin registration server" Jul 15 10:19:58.264521 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.264508 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 10:19:58.264572 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.264522 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 10:19:58.264634 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.264613 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 10:19:58.264899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.264690 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 10:19:58.264899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.264702 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 10:19:58.265240 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.265217 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 10:19:58.265328 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.265274 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:58.343660 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.343631 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 10:19:58.344877 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.344862 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 10:19:58.344938 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.344897 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 10:19:58.344938 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.344915 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 10:19:58.345016 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.344948 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 10:19:58.347297 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.347282 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:19:58.365563 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.365544 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:19:58.366408 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.366396 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:19:58.366468 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.366421 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:19:58.366468 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.366433 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:19:58.366468 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.366460 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.379053 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.379031 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.379053 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.379050 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-247.ec2.internal\": node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:58.396927 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.396883 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:58.445850 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.445830 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal"] Jul 15 10:19:58.445914 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.445897 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:19:58.446707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.446693 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:19:58.446791 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.446719 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:19:58.446791 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.446765 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:19:58.448228 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.448215 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:19:58.448345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.448331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.448402 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.448358 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:19:58.448930 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.448913 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:19:58.448930 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.448927 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:19:58.449059 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.448942 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:19:58.449059 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.448946 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:19:58.449059 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.448954 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:19:58.449059 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.448963 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:19:58.450104 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.450090 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.450158 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.450115 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:19:58.450715 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.450699 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:19:58.450830 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.450723 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:19:58.450830 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.450736 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:19:58.476248 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.476224 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-247.ec2.internal\" not found" node="ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.480489 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.480473 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-247.ec2.internal\" not found" node="ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.496962 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.496945 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:58.512403 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.512385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/740d9dfc5ebbe2db89963886b2766aa3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-247.ec2.internal\" (UID: \"740d9dfc5ebbe2db89963886b2766aa3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.512452 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.512410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89fab6746b992e432b20b770a46b339d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal\" (UID: \"89fab6746b992e432b20b770a46b339d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.512452 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.512427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89fab6746b992e432b20b770a46b339d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal\" (UID: \"89fab6746b992e432b20b770a46b339d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.597650 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.597622 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:58.613131 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.613110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89fab6746b992e432b20b770a46b339d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal\" (UID: \"89fab6746b992e432b20b770a46b339d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.613181 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.613142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89fab6746b992e432b20b770a46b339d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal\" (UID: \"89fab6746b992e432b20b770a46b339d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.613181 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.613158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/740d9dfc5ebbe2db89963886b2766aa3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-247.ec2.internal\" (UID: \"740d9dfc5ebbe2db89963886b2766aa3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.613258 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.613198 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89fab6746b992e432b20b770a46b339d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal\" (UID: \"89fab6746b992e432b20b770a46b339d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.613258 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.613227 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89fab6746b992e432b20b770a46b339d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal\" (UID: \"89fab6746b992e432b20b770a46b339d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.613258 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.613204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/740d9dfc5ebbe2db89963886b2766aa3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-247.ec2.internal\" (UID: \"740d9dfc5ebbe2db89963886b2766aa3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.698586 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.698523 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:58.779033 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.779014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.783892 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:58.783871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal" Jul 15 10:19:58.799381 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.799348 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:58.899903 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:58.899877 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:59.000704 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:59.000676 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:59.100785 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:59.100756 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:59.151768 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.151729 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 10:19:59.152238 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.151884 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 10:19:59.200923 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:59.200900 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:59.223484 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.223414 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 10:14:58 +0000 UTC" deadline="2028-04-29 22:56:17.14343231 +0000 UTC" Jul 15 10:19:59.223484 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.223447 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15708h36m17.919988284s" Jul 15 10:19:59.271876 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:59.271840 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fab6746b992e432b20b770a46b339d.slice/crio-5e8b49a4aea07fc82877122a27b973fcced24f0cf27588e9dfda0a98d2e5a5b8 WatchSource:0}: Error finding container 5e8b49a4aea07fc82877122a27b973fcced24f0cf27588e9dfda0a98d2e5a5b8: Status 404 returned error can't find the container with id 5e8b49a4aea07fc82877122a27b973fcced24f0cf27588e9dfda0a98d2e5a5b8 Jul 15 10:19:59.272381 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:19:59.272359 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740d9dfc5ebbe2db89963886b2766aa3.slice/crio-dc45967b0a7d670362096310211be3f4b6157002f39f0c687e4d5a0fb4741cf8 WatchSource:0}: Error finding container dc45967b0a7d670362096310211be3f4b6157002f39f0c687e4d5a0fb4741cf8: Status 404 returned error can't find the container with id dc45967b0a7d670362096310211be3f4b6157002f39f0c687e4d5a0fb4741cf8 Jul 15 10:19:59.277042 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.277026 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 10:19:59.301781 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:19:59.301738 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-247.ec2.internal\" not found" Jul 15 10:19:59.348123 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.348083 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal" event={"ID":"740d9dfc5ebbe2db89963886b2766aa3","Type":"ContainerStarted","Data":"dc45967b0a7d670362096310211be3f4b6157002f39f0c687e4d5a0fb4741cf8"} Jul 15 10:19:59.349211 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.349192 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" event={"ID":"89fab6746b992e432b20b770a46b339d","Type":"ContainerStarted","Data":"5e8b49a4aea07fc82877122a27b973fcced24f0cf27588e9dfda0a98d2e5a5b8"} Jul 15 10:19:59.364368 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.364344 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:19:59.411703 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.411681 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" Jul 15 10:19:59.424059 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.424043 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 10:19:59.426003 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.425990 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal" Jul 15 10:19:59.433979 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.433956 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 10:19:59.445183 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.445163 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:19:59.610970 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:19:59.610909 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:20:00.190964 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.190918 2579 apiserver.go:52] "Watching apiserver" Jul 15 10:20:00.200541 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.200511 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 10:20:00.200952 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.200924 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal","openshift-network-diagnostics/network-check-target-5ppb9","openshift-network-operator/iptables-alerter-97k4j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2","openshift-dns/node-resolver-q95hm","openshift-multus/multus-758tc","openshift-multus/multus-additional-cni-plugins-xhxtz","openshift-multus/network-metrics-daemon-cp4gp","openshift-ovn-kubernetes/ovnkube-node-klwfw","kube-system/global-pull-secret-syncer-mw4m6","kube-system/konnectivity-agent-9dvfc","kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal","openshift-cluster-node-tuning-operator/tuned-ssrgb","openshift-image-registry/node-ca-zn4vg"] Jul 15 10:20:00.202596 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.202574 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.203911 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.203887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.205108 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.205088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.205211 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.205167 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lx6bh\"" Jul 15 10:20:00.205809 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.205788 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 10:20:00.205935 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.205839 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 10:20:00.205935 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.205790 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 10:20:00.205935 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.205789 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 10:20:00.206083 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.205957 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 10:20:00.206687 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.206667 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 10:20:00.206687 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.206677 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rgg87\"" Jul 15 10:20:00.206899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.206690 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 10:20:00.206958 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.206940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.207655 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.207636 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 10:20:00.208258 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.208238 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 10:20:00.208368 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.208289 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 10:20:00.208429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.208396 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.208484 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.208240 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gn9h8\"" Jul 15 10:20:00.213170 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.210312 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 10:20:00.213170 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.211252 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-txv7z\"" Jul 15 10:20:00.213170 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.211357 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-skvt5\"" Jul 15 10:20:00.213170 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.211384 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 10:20:00.213170 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.211686 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 10:20:00.213170 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.211913 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 10:20:00.213170 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.210314 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 10:20:00.213170 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.212592 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-758tc" Jul 15 10:20:00.213170 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.212620 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 10:20:00.214024 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.213252 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.216302 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.215163 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 10:20:00.216302 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.215775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.216302 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.215875 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g4lzv\"" Jul 15 10:20:00.216854 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.216818 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c94h4\"" Jul 15 10:20:00.217047 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.217017 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 10:20:00.217229 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.217203 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 10:20:00.217633 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.217609 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 10:20:00.217715 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.217638 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 10:20:00.217988 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.217969 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:00.218113 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.218010 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 10:20:00.218113 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.217974 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 10:20:00.218113 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.218053 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:00.218695 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.218672 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 10:20:00.218897 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.218883 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 10:20:00.219390 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.219373 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hx8sx\"" Jul 15 10:20:00.219518 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.219502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:00.219613 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.219570 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:00.219767 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.219702 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 10:20:00.220648 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.220546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:00.221481 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.221463 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:00.221591 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.221517 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:00.222701 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.222682 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 10:20:00.223206 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.223005 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 10:20:00.223206 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.223009 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p44d5\"" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.223857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-systemd\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.223893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-run\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.223925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-var-lib-kubelet\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.223951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.223974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0471e5a-45bf-4183-b286-8904049cb8ce-hosts-file\") pod \"node-resolver-q95hm\" (UID: \"e0471e5a-45bf-4183-b286-8904049cb8ce\") " pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-var-lib-cni-bin\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxzf\" (UniqueName: \"kubernetes.io/projected/15dab779-ba59-448b-9a1a-6e7e28e5dec9-kube-api-access-jmxzf\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-tuned\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-systemd-units\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-run-openvswitch\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/492b2069-1e55-477f-b333-19f21a28a3f0-ovn-node-metrics-cert\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-system-cni-dir\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2da181a5-da16-4f42-b591-6aef7ed637bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqqs\" (UniqueName: \"kubernetes.io/projected/2da181a5-da16-4f42-b591-6aef7ed637bc-kube-api-access-fdqqs\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-cnibin\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.224969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-run-ovn\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-sys\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-os-release\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-daemon-config\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224411 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-slash\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-var-lib-openvswitch\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-node-log\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-sysctl-d\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-socket-dir-parent\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-run-k8s-cni-cncf-io\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-cni-bin\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15dab779-ba59-448b-9a1a-6e7e28e5dec9-tmp\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79f325cb-b2b5-4788-9fc6-83b05c671817-host-slash\") pod \"iptables-alerter-97k4j\" (UID: \"79f325cb-b2b5-4788-9fc6-83b05c671817\") " pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-var-lib-kubelet\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-conf-dir\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-run-netns\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.225686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-run-ovn-kubernetes\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-sysconfig\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb7bb\" (UniqueName: \"kubernetes.io/projected/2fd275ac-6422-49df-a98e-ae1756a0e2c8-kube-api-access-fb7bb\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-system-cni-dir\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-os-release\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/492b2069-1e55-477f-b333-19f21a28a3f0-ovnkube-config\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.224985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/492b2069-1e55-477f-b333-19f21a28a3f0-ovnkube-script-lib\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5665h\" (UniqueName: \"kubernetes.io/projected/492b2069-1e55-477f-b333-19f21a28a3f0-kube-api-access-5665h\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225060 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-cni-dir\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-run-netns\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-kubelet\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-kubernetes\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-socket-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0471e5a-45bf-4183-b286-8904049cb8ce-tmp-dir\") pod \"node-resolver-q95hm\" (UID: \"e0471e5a-45bf-4183-b286-8904049cb8ce\") " pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-hostroot\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-etc-kubernetes\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225317 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/492b2069-1e55-477f-b333-19f21a28a3f0-env-overrides\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.226345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-lib-modules\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-registration-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-device-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfnw8\" (UniqueName: \"kubernetes.io/projected/79f325cb-b2b5-4788-9fc6-83b05c671817-kube-api-access-dfnw8\") pod \"iptables-alerter-97k4j\" (UID: \"79f325cb-b2b5-4788-9fc6-83b05c671817\") " pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2da181a5-da16-4f42-b591-6aef7ed637bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-cnibin\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79f325cb-b2b5-4788-9fc6-83b05c671817-iptables-alerter-script\") pod \"iptables-alerter-97k4j\" (UID: \"79f325cb-b2b5-4788-9fc6-83b05c671817\") " pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-sys-fs\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225551 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-etc-openvswitch\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-log-socket\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225634 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-modprobe-d\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-sysctl-conf\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-host\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-run-multus-certs\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2da181a5-da16-4f42-b591-6aef7ed637bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.227097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-cni-netd\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.227749 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.227749 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2023908-01cc-4632-86d3-3d4f3b18e4f2-cni-binary-copy\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.227749 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-run-systemd\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.227749 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487dx\" (UniqueName: \"kubernetes.io/projected/e0471e5a-45bf-4183-b286-8904049cb8ce-kube-api-access-487dx\") pod \"node-resolver-q95hm\" (UID: \"e0471e5a-45bf-4183-b286-8904049cb8ce\") " pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.227749 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-var-lib-cni-multus\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.227749 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.225993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98bl\" (UniqueName: \"kubernetes.io/projected/a2023908-01cc-4632-86d3-3d4f3b18e4f2-kube-api-access-c98bl\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.227749 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.226896 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 10:20:00.255604 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.255574 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c2tjf" Jul 15 10:20:00.265756 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.265702 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c2tjf" Jul 15 10:20:00.312429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.312400 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 10:20:00.327220 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqqs\" (UniqueName: \"kubernetes.io/projected/2da181a5-da16-4f42-b591-6aef7ed637bc-kube-api-access-fdqqs\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.327392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-cnibin\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.327392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-run-ovn\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.327392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-sys\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.327392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-os-release\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.327392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-daemon-config\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.327392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327373 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-cnibin\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.327674 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327419 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-os-release\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.327674 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-run-ovn\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.327674 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-sys\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.327674 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:00.327674 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssf6n\" (UniqueName: \"kubernetes.io/projected/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-kube-api-access-ssf6n\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:00.327674 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e068402-7f79-4214-b488-5ee985a633ec-agent-certs\") pod \"konnectivity-agent-9dvfc\" (UID: \"3e068402-7f79-4214-b488-5ee985a633ec\") " pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:00.327674 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-slash\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.328061 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-var-lib-openvswitch\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.328061 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-node-log\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.328061 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-slash\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.328061 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-sysctl-d\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.328061 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-var-lib-openvswitch\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.328061 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.327858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-node-log\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.328061 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328032 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-sysctl-d\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.328415 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328086 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-daemon-config\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.328415 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-socket-dir-parent\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.328415 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-run-k8s-cni-cncf-io\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.328415 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-cni-bin\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.328415 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15dab779-ba59-448b-9a1a-6e7e28e5dec9-tmp\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79f325cb-b2b5-4788-9fc6-83b05c671817-host-slash\") pod \"iptables-alerter-97k4j\" (UID: \"79f325cb-b2b5-4788-9fc6-83b05c671817\") " pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328463 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-run-k8s-cni-cncf-io\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79f325cb-b2b5-4788-9fc6-83b05c671817-host-slash\") pod \"iptables-alerter-97k4j\" (UID: \"79f325cb-b2b5-4788-9fc6-83b05c671817\") " pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328492 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-cni-bin\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328560 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-socket-dir-parent\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-var-lib-kubelet\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-conf-dir\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-var-lib-kubelet\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.328664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-run-netns\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-run-ovn-kubernetes\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-run-netns\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-conf-dir\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-sysconfig\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb7bb\" (UniqueName: \"kubernetes.io/projected/2fd275ac-6422-49df-a98e-ae1756a0e2c8-kube-api-access-fb7bb\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328807 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-run-ovn-kubernetes\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-system-cni-dir\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-sysconfig\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-kubelet-config\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328843 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-os-release\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.328977 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-system-cni-dir\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/492b2069-1e55-477f-b333-19f21a28a3f0-ovnkube-config\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-os-release\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/492b2069-1e55-477f-b333-19f21a28a3f0-ovnkube-script-lib\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5665h\" (UniqueName: \"kubernetes.io/projected/492b2069-1e55-477f-b333-19f21a28a3f0-kube-api-access-5665h\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-cni-dir\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.329215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-run-netns\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-run-netns\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329280 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-multus-cni-dir\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-kubelet\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-kubernetes\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-socket-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0471e5a-45bf-4183-b286-8904049cb8ce-tmp-dir\") pod \"node-resolver-q95hm\" (UID: \"e0471e5a-45bf-4183-b286-8904049cb8ce\") " pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-kubelet\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-hostroot\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-kubernetes\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329476 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-hostroot\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-etc-kubernetes\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/492b2069-1e55-477f-b333-19f21a28a3f0-env-overrides\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-lib-modules\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-etc-kubernetes\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-registration-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-device-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfnw8\" (UniqueName: \"kubernetes.io/projected/79f325cb-b2b5-4788-9fc6-83b05c671817-kube-api-access-dfnw8\") pod \"iptables-alerter-97k4j\" (UID: \"79f325cb-b2b5-4788-9fc6-83b05c671817\") " pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.330133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-socket-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2da181a5-da16-4f42-b591-6aef7ed637bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-cnibin\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvr87\" (UniqueName: \"kubernetes.io/projected/cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2-kube-api-access-zvr87\") pod \"node-ca-zn4vg\" (UID: \"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2\") " pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79f325cb-b2b5-4788-9fc6-83b05c671817-iptables-alerter-script\") pod \"iptables-alerter-97k4j\" (UID: \"79f325cb-b2b5-4788-9fc6-83b05c671817\") " pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-sys-fs\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-etc-openvswitch\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-log-socket\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-modprobe-d\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-sysctl-conf\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-host\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-run-multus-certs\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2da181a5-da16-4f42-b591-6aef7ed637bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2-host\") pod \"node-ca-zn4vg\" (UID: \"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2\") " pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/492b2069-1e55-477f-b333-19f21a28a3f0-ovnkube-config\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-cni-netd\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.330773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-cni-netd\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.331314 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330822 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-device-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.331314 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.330939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/492b2069-1e55-477f-b333-19f21a28a3f0-env-overrides\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.331314 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.329770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-lib-modules\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.331314 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331043 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0471e5a-45bf-4183-b286-8904049cb8ce-tmp-dir\") pod \"node-resolver-q95hm\" (UID: \"e0471e5a-45bf-4183-b286-8904049cb8ce\") " pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.331314 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331229 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/492b2069-1e55-477f-b333-19f21a28a3f0-ovnkube-script-lib\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.331314 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-registration-dir\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.331632 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2da181a5-da16-4f42-b591-6aef7ed637bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.331632 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-modprobe-d\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.331703 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-cnibin\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.331923 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-sysctl-conf\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.331923 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-etc-openvswitch\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.331923 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-run-multus-certs\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.331923 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-sys-fs\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.331923 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-log-socket\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.332223 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.332223 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.331955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-host\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.332223 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2-serviceca\") pod \"node-ca-zn4vg\" (UID: \"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2\") " pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.332223 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppnq6\" (UniqueName: \"kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6\") pod \"network-check-target-5ppb9\" (UID: \"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2\") " pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:00.332223 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-dbus\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:00.332223 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e068402-7f79-4214-b488-5ee985a633ec-konnectivity-ca\") pod \"konnectivity-agent-9dvfc\" (UID: \"3e068402-7f79-4214-b488-5ee985a633ec\") " pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:00.332523 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.332523 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2023908-01cc-4632-86d3-3d4f3b18e4f2-cni-binary-copy\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.332523 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332362 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79f325cb-b2b5-4788-9fc6-83b05c671817-iptables-alerter-script\") pod \"iptables-alerter-97k4j\" (UID: \"79f325cb-b2b5-4788-9fc6-83b05c671817\") " pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.332523 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:00.332716 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332538 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.332716 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-run-systemd\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.332716 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487dx\" (UniqueName: \"kubernetes.io/projected/e0471e5a-45bf-4183-b286-8904049cb8ce-kube-api-access-487dx\") pod \"node-resolver-q95hm\" (UID: \"e0471e5a-45bf-4183-b286-8904049cb8ce\") " pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.332890 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-var-lib-cni-multus\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.332890 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98bl\" (UniqueName: \"kubernetes.io/projected/a2023908-01cc-4632-86d3-3d4f3b18e4f2-kube-api-access-c98bl\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.332890 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-systemd\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.333089 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.332969 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2023908-01cc-4632-86d3-3d4f3b18e4f2-cni-binary-copy\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.333089 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-run\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.333184 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-var-lib-kubelet\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.333184 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.333280 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0471e5a-45bf-4183-b286-8904049cb8ce-hosts-file\") pod \"node-resolver-q95hm\" (UID: \"e0471e5a-45bf-4183-b286-8904049cb8ce\") " pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.333330 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-var-lib-cni-bin\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.333330 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333307 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-var-lib-cni-multus\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.333330 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxzf\" (UniqueName: \"kubernetes.io/projected/15dab779-ba59-448b-9a1a-6e7e28e5dec9-kube-api-access-jmxzf\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.333484 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-tuned\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.333484 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-run-systemd\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.333484 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333465 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-var-lib-kubelet\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.333484 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333472 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15dab779-ba59-448b-9a1a-6e7e28e5dec9-tmp\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.333681 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0471e5a-45bf-4183-b286-8904049cb8ce-hosts-file\") pod \"node-resolver-q95hm\" (UID: \"e0471e5a-45bf-4183-b286-8904049cb8ce\") " pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.333681 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333594 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2fd275ac-6422-49df-a98e-ae1756a0e2c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.333681 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-systemd\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.333856 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-systemd-units\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.333856 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333768 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2023908-01cc-4632-86d3-3d4f3b18e4f2-host-var-lib-cni-bin\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.333966 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15dab779-ba59-448b-9a1a-6e7e28e5dec9-run\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.334019 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.333981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-run-openvswitch\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.334070 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.334047 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-systemd-units\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.334128 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.334087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/492b2069-1e55-477f-b333-19f21a28a3f0-ovn-node-metrics-cert\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.334185 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.334165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/492b2069-1e55-477f-b333-19f21a28a3f0-run-openvswitch\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.334242 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.334221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-system-cni-dir\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.334375 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.334352 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2da181a5-da16-4f42-b591-6aef7ed637bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.334452 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.334347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2da181a5-da16-4f42-b591-6aef7ed637bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.334514 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.334503 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2da181a5-da16-4f42-b591-6aef7ed637bc-system-cni-dir\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.335152 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.335124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2da181a5-da16-4f42-b591-6aef7ed637bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.337078 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.336958 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15dab779-ba59-448b-9a1a-6e7e28e5dec9-etc-tuned\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.337464 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.337438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqqs\" (UniqueName: \"kubernetes.io/projected/2da181a5-da16-4f42-b591-6aef7ed637bc-kube-api-access-fdqqs\") pod \"multus-additional-cni-plugins-xhxtz\" (UID: \"2da181a5-da16-4f42-b591-6aef7ed637bc\") " pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.338527 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.338497 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5665h\" (UniqueName: \"kubernetes.io/projected/492b2069-1e55-477f-b333-19f21a28a3f0-kube-api-access-5665h\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.339693 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.339647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb7bb\" (UniqueName: \"kubernetes.io/projected/2fd275ac-6422-49df-a98e-ae1756a0e2c8-kube-api-access-fb7bb\") pod \"aws-ebs-csi-driver-node-tqgp2\" (UID: \"2fd275ac-6422-49df-a98e-ae1756a0e2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.340334 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.340308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfnw8\" (UniqueName: \"kubernetes.io/projected/79f325cb-b2b5-4788-9fc6-83b05c671817-kube-api-access-dfnw8\") pod \"iptables-alerter-97k4j\" (UID: \"79f325cb-b2b5-4788-9fc6-83b05c671817\") " pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.341111 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.341086 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/492b2069-1e55-477f-b333-19f21a28a3f0-ovn-node-metrics-cert\") pod \"ovnkube-node-klwfw\" (UID: \"492b2069-1e55-477f-b333-19f21a28a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.344661 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.344638 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-487dx\" (UniqueName: \"kubernetes.io/projected/e0471e5a-45bf-4183-b286-8904049cb8ce-kube-api-access-487dx\") pod \"node-resolver-q95hm\" (UID: \"e0471e5a-45bf-4183-b286-8904049cb8ce\") " pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.345425 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.345402 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxzf\" (UniqueName: \"kubernetes.io/projected/15dab779-ba59-448b-9a1a-6e7e28e5dec9-kube-api-access-jmxzf\") pod \"tuned-ssrgb\" (UID: \"15dab779-ba59-448b-9a1a-6e7e28e5dec9\") " pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.345725 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.345707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98bl\" (UniqueName: \"kubernetes.io/projected/a2023908-01cc-4632-86d3-3d4f3b18e4f2-kube-api-access-c98bl\") pod \"multus-758tc\" (UID: \"a2023908-01cc-4632-86d3-3d4f3b18e4f2\") " pod="openshift-multus/multus-758tc" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.434744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.434815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssf6n\" (UniqueName: \"kubernetes.io/projected/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-kube-api-access-ssf6n\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.434842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e068402-7f79-4214-b488-5ee985a633ec-agent-certs\") pod \"konnectivity-agent-9dvfc\" (UID: \"3e068402-7f79-4214-b488-5ee985a633ec\") " pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.434895 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.434960 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-kubelet-config\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.434983 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs podName:81be95be-8bfa-45cb-b2d6-30058fbf2e3b nodeName:}" failed. No retries permitted until 2026-07-15 10:20:00.934957933 +0000 UTC m=+3.068151269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs") pod "network-metrics-daemon-cp4gp" (UID: "81be95be-8bfa-45cb-b2d6-30058fbf2e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.434895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-kubelet-config\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvr87\" (UniqueName: \"kubernetes.io/projected/cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2-kube-api-access-zvr87\") pod \"node-ca-zn4vg\" (UID: \"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2\") " pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2-host\") pod \"node-ca-zn4vg\" (UID: \"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2\") " pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2-serviceca\") pod \"node-ca-zn4vg\" (UID: \"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2\") " pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnq6\" (UniqueName: \"kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6\") pod \"network-check-target-5ppb9\" (UID: \"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2\") " pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435127 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2-host\") pod \"node-ca-zn4vg\" (UID: \"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2\") " pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-dbus\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e068402-7f79-4214-b488-5ee985a633ec-konnectivity-ca\") pod \"konnectivity-agent-9dvfc\" (UID: \"3e068402-7f79-4214-b488-5ee985a633ec\") " pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.435300 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:00.435451 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.435361 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret podName:4fd5ac79-bd46-4d5e-9c77-9df73609b5b8 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:00.935343128 +0000 UTC m=+3.068536461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret") pod "global-pull-secret-syncer-mw4m6" (UID: "4fd5ac79-bd46-4d5e-9c77-9df73609b5b8") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:00.436379 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-dbus\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:00.436379 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2-serviceca\") pod \"node-ca-zn4vg\" (UID: \"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2\") " pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.436379 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.435821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e068402-7f79-4214-b488-5ee985a633ec-konnectivity-ca\") pod \"konnectivity-agent-9dvfc\" (UID: \"3e068402-7f79-4214-b488-5ee985a633ec\") " pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:00.438137 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.438111 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e068402-7f79-4214-b488-5ee985a633ec-agent-certs\") pod \"konnectivity-agent-9dvfc\" (UID: \"3e068402-7f79-4214-b488-5ee985a633ec\") " pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:00.444283 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.444209 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:20:00.444283 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.444236 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:20:00.444283 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.444252 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppnq6 for pod openshift-network-diagnostics/network-check-target-5ppb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:00.444503 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.444327 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6 podName:1b1d839b-8804-451e-b28b-ecf0f7a5e5f2 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:00.944308174 +0000 UTC m=+3.077501506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ppnq6" (UniqueName: "kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6") pod "network-check-target-5ppb9" (UID: "1b1d839b-8804-451e-b28b-ecf0f7a5e5f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:00.447670 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.447642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssf6n\" (UniqueName: \"kubernetes.io/projected/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-kube-api-access-ssf6n\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:00.447829 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.447638 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvr87\" (UniqueName: \"kubernetes.io/projected/cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2-kube-api-access-zvr87\") pod \"node-ca-zn4vg\" (UID: \"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2\") " pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.497700 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.497668 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:20:00.517285 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.517247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" Jul 15 10:20:00.524781 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.524750 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" Jul 15 10:20:00.535832 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.535811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-97k4j" Jul 15 10:20:00.540800 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.540772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" Jul 15 10:20:00.547499 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.547481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q95hm" Jul 15 10:20:00.554496 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.554474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-758tc" Jul 15 10:20:00.560726 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.560710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:00.567907 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.567887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zn4vg" Jul 15 10:20:00.573630 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.573608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:00.939879 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.939846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:00.940039 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:00.939899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:00.940039 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.940018 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:00.940039 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.940026 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:00.940190 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.940091 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs podName:81be95be-8bfa-45cb-b2d6-30058fbf2e3b nodeName:}" failed. No retries permitted until 2026-07-15 10:20:01.940071163 +0000 UTC m=+4.073264498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs") pod "network-metrics-daemon-cp4gp" (UID: "81be95be-8bfa-45cb-b2d6-30058fbf2e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:00.940190 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:00.940114 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret podName:4fd5ac79-bd46-4d5e-9c77-9df73609b5b8 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:01.940102561 +0000 UTC m=+4.073295880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret") pod "global-pull-secret-syncer-mw4m6" (UID: "4fd5ac79-bd46-4d5e-9c77-9df73609b5b8") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:01.040629 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.040505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnq6\" (UniqueName: \"kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6\") pod \"network-check-target-5ppb9\" (UID: \"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2\") " pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:01.040838 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:01.040699 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:20:01.040838 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:01.040733 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:20:01.040838 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:01.040750 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppnq6 for pod openshift-network-diagnostics/network-check-target-5ppb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:01.041003 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:01.040849 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6 podName:1b1d839b-8804-451e-b28b-ecf0f7a5e5f2 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:02.040824771 +0000 UTC m=+4.174018110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppnq6" (UniqueName: "kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6") pod "network-check-target-5ppb9" (UID: "1b1d839b-8804-451e-b28b-ecf0f7a5e5f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:01.228817 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:20:01.228773 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f325cb_b2b5_4788_9fc6_83b05c671817.slice/crio-5ac6ca8e70af1a5f2e7827946b32c035aac7ba38b6bb6b87529eeb6a41190d09 WatchSource:0}: Error finding container 5ac6ca8e70af1a5f2e7827946b32c035aac7ba38b6bb6b87529eeb6a41190d09: Status 404 returned error can't find the container with id 5ac6ca8e70af1a5f2e7827946b32c035aac7ba38b6bb6b87529eeb6a41190d09 Jul 15 10:20:01.236973 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:20:01.236929 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0471e5a_45bf_4183_b286_8904049cb8ce.slice/crio-6807b6ea6e71df7be5342f4e0205f0f65db194464e7092a370ddea4f2a54fb1e WatchSource:0}: Error finding container 6807b6ea6e71df7be5342f4e0205f0f65db194464e7092a370ddea4f2a54fb1e: Status 404 returned error can't find the container with id 6807b6ea6e71df7be5342f4e0205f0f65db194464e7092a370ddea4f2a54fb1e Jul 15 10:20:01.237537 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:20:01.237516 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da181a5_da16_4f42_b591_6aef7ed637bc.slice/crio-16ee21e153b92ed76b87ed38c888d3ab8ec0d57a11ba52489f20c417064432a2 WatchSource:0}: Error finding container 16ee21e153b92ed76b87ed38c888d3ab8ec0d57a11ba52489f20c417064432a2: Status 404 returned error can't find the container with id 16ee21e153b92ed76b87ed38c888d3ab8ec0d57a11ba52489f20c417064432a2 Jul 15 10:20:01.242322 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:20:01.239244 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd275ac_6422_49df_a98e_ae1756a0e2c8.slice/crio-3ace80b5a22fc6735783ee6c635adc9d24e72166273581bac49cc11054a82620 WatchSource:0}: Error finding container 3ace80b5a22fc6735783ee6c635adc9d24e72166273581bac49cc11054a82620: Status 404 returned error can't find the container with id 3ace80b5a22fc6735783ee6c635adc9d24e72166273581bac49cc11054a82620 Jul 15 10:20:01.268353 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.268175 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 10:15:00 +0000 UTC" deadline="2028-04-10 07:42:11.890492712 +0000 UTC" Jul 15 10:20:01.268353 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.268353 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15237h22m10.622143892s" Jul 15 10:20:01.345940 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.345910 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:01.346071 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:01.346023 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:01.355165 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.355124 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" event={"ID":"492b2069-1e55-477f-b333-19f21a28a3f0","Type":"ContainerStarted","Data":"40eb157f29d34a1e74482d729cd2dd0fd454e0d688cf4eff27199081ac1fdfe3"} Jul 15 10:20:01.357696 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.357673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" event={"ID":"15dab779-ba59-448b-9a1a-6e7e28e5dec9","Type":"ContainerStarted","Data":"2ac551cd3a7c4e097673423604bdff28c8d10c82d40a998b0c95e6eee166a5b6"} Jul 15 10:20:01.359207 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.359186 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" event={"ID":"2fd275ac-6422-49df-a98e-ae1756a0e2c8","Type":"ContainerStarted","Data":"3ace80b5a22fc6735783ee6c635adc9d24e72166273581bac49cc11054a82620"} Jul 15 10:20:01.360273 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.360252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zn4vg" event={"ID":"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2","Type":"ContainerStarted","Data":"e52c41c6943e0917798031c6e2045d3c8bd7a9e1cf5df3901407c0b55ae02987"} Jul 15 10:20:01.361310 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.361292 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-758tc" event={"ID":"a2023908-01cc-4632-86d3-3d4f3b18e4f2","Type":"ContainerStarted","Data":"69f205e58df86b407a29683245ad71f758cc8e92c7e7f8b780da4be17e193a8b"} Jul 15 10:20:01.364133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.364108 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" event={"ID":"2da181a5-da16-4f42-b591-6aef7ed637bc","Type":"ContainerStarted","Data":"16ee21e153b92ed76b87ed38c888d3ab8ec0d57a11ba52489f20c417064432a2"} Jul 15 10:20:01.365316 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.365294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9dvfc" event={"ID":"3e068402-7f79-4214-b488-5ee985a633ec","Type":"ContainerStarted","Data":"25a692f7c69c49e2cb5a80a22ae0f571c0df55c4a0757d3f046e6203ea633cb5"} Jul 15 10:20:01.367801 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.367780 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q95hm" event={"ID":"e0471e5a-45bf-4183-b286-8904049cb8ce","Type":"ContainerStarted","Data":"6807b6ea6e71df7be5342f4e0205f0f65db194464e7092a370ddea4f2a54fb1e"} Jul 15 10:20:01.371172 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.371151 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-97k4j" event={"ID":"79f325cb-b2b5-4788-9fc6-83b05c671817","Type":"ContainerStarted","Data":"5ac6ca8e70af1a5f2e7827946b32c035aac7ba38b6bb6b87529eeb6a41190d09"} Jul 15 10:20:01.373045 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.373025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal" event={"ID":"740d9dfc5ebbe2db89963886b2766aa3","Type":"ContainerStarted","Data":"7844fb2edd2ae500fdd1ccaf4f46760f24109a80ad464e7b30631520e57284ec"} Jul 15 10:20:01.396055 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.396015 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-247.ec2.internal" podStartSLOduration=2.396000377 podStartE2EDuration="2.396000377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:01.395946447 +0000 UTC m=+3.529139790" watchObservedRunningTime="2026-07-15 10:20:01.396000377 +0000 UTC m=+3.529193721" Jul 15 10:20:01.947074 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.946983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:01.947074 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:01.947047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:01.947292 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:01.947176 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:01.947292 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:01.947192 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:01.947292 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:01.947250 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs podName:81be95be-8bfa-45cb-b2d6-30058fbf2e3b nodeName:}" failed. No retries permitted until 2026-07-15 10:20:03.947230223 +0000 UTC m=+6.080423545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs") pod "network-metrics-daemon-cp4gp" (UID: "81be95be-8bfa-45cb-b2d6-30058fbf2e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:01.947292 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:01.947269 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret podName:4fd5ac79-bd46-4d5e-9c77-9df73609b5b8 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:03.947259072 +0000 UTC m=+6.080452391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret") pod "global-pull-secret-syncer-mw4m6" (UID: "4fd5ac79-bd46-4d5e-9c77-9df73609b5b8") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:02.047394 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:02.047362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnq6\" (UniqueName: \"kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6\") pod \"network-check-target-5ppb9\" (UID: \"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2\") " pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:02.047585 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:02.047517 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:20:02.047585 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:02.047533 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:20:02.047585 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:02.047542 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppnq6 for pod openshift-network-diagnostics/network-check-target-5ppb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:02.047757 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:02.047591 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6 podName:1b1d839b-8804-451e-b28b-ecf0f7a5e5f2 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:04.04757671 +0000 UTC m=+6.180770027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppnq6" (UniqueName: "kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6") pod "network-check-target-5ppb9" (UID: "1b1d839b-8804-451e-b28b-ecf0f7a5e5f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:02.269039 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:02.268910 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 10:15:00 +0000 UTC" deadline="2028-03-07 14:46:43.203902987 +0000 UTC" Jul 15 10:20:02.269039 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:02.268947 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14428h26m40.934960513s" Jul 15 10:20:02.349745 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:02.348714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:02.349745 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:02.348880 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:02.349745 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:02.349316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:02.349745 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:02.349416 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:02.387521 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:02.387487 2579 generic.go:356] "Generic (PLEG): container finished" podID="89fab6746b992e432b20b770a46b339d" containerID="97060caf7fae4fdee1000f21cc474c7d2cd3db2cbc447be33836e07d01b9556e" exitCode=0 Jul 15 10:20:02.388444 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:02.388417 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" event={"ID":"89fab6746b992e432b20b770a46b339d","Type":"ContainerDied","Data":"97060caf7fae4fdee1000f21cc474c7d2cd3db2cbc447be33836e07d01b9556e"} Jul 15 10:20:03.345177 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:03.345144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:03.345625 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:03.345269 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:03.403311 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:03.403274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" event={"ID":"89fab6746b992e432b20b770a46b339d","Type":"ContainerStarted","Data":"552f05156a05d7cb84d02658237c89a202d0002b31859aa0d15557be63a0dc42"} Jul 15 10:20:03.425048 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:03.424998 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-247.ec2.internal" podStartSLOduration=4.424978605 podStartE2EDuration="4.424978605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:03.423999736 +0000 UTC m=+5.557193268" watchObservedRunningTime="2026-07-15 10:20:03.424978605 +0000 UTC m=+5.558171943" Jul 15 10:20:03.964471 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:03.963662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:03.964471 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:03.963724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:03.964471 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:03.963907 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:03.964471 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:03.963972 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs podName:81be95be-8bfa-45cb-b2d6-30058fbf2e3b nodeName:}" failed. No retries permitted until 2026-07-15 10:20:07.963954023 +0000 UTC m=+10.097147341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs") pod "network-metrics-daemon-cp4gp" (UID: "81be95be-8bfa-45cb-b2d6-30058fbf2e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:03.964471 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:03.964376 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:03.964471 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:03.964424 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret podName:4fd5ac79-bd46-4d5e-9c77-9df73609b5b8 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:07.964409053 +0000 UTC m=+10.097602372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret") pod "global-pull-secret-syncer-mw4m6" (UID: "4fd5ac79-bd46-4d5e-9c77-9df73609b5b8") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:04.065185 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:04.065103 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnq6\" (UniqueName: \"kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6\") pod \"network-check-target-5ppb9\" (UID: \"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2\") " pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:04.065356 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:04.065311 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:20:04.065356 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:04.065333 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:20:04.065356 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:04.065345 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppnq6 for pod openshift-network-diagnostics/network-check-target-5ppb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:04.065507 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:04.065402 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6 podName:1b1d839b-8804-451e-b28b-ecf0f7a5e5f2 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:08.065384451 +0000 UTC m=+10.198577783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppnq6" (UniqueName: "kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6") pod "network-check-target-5ppb9" (UID: "1b1d839b-8804-451e-b28b-ecf0f7a5e5f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:04.349150 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:04.348399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:04.349150 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:04.348554 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:04.349150 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:04.348961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:04.349150 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:04.349055 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:05.345697 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:05.345316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:05.345697 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:05.345470 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:06.348262 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:06.346850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:06.348262 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:06.346989 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:06.348262 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:06.347478 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:06.348262 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:06.347576 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:07.345156 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:07.345121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:07.345314 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:07.345245 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:07.999077 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:07.998507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:07.999077 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:07.998572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:07.999077 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:07.998679 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:07.999077 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:07.998678 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:07.999077 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:07.998731 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs podName:81be95be-8bfa-45cb-b2d6-30058fbf2e3b nodeName:}" failed. No retries permitted until 2026-07-15 10:20:15.998717201 +0000 UTC m=+18.131910518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs") pod "network-metrics-daemon-cp4gp" (UID: "81be95be-8bfa-45cb-b2d6-30058fbf2e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:07.999077 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:07.998783 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret podName:4fd5ac79-bd46-4d5e-9c77-9df73609b5b8 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:15.99876306 +0000 UTC m=+18.131956376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret") pod "global-pull-secret-syncer-mw4m6" (UID: "4fd5ac79-bd46-4d5e-9c77-9df73609b5b8") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:08.099785 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:08.099735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnq6\" (UniqueName: \"kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6\") pod \"network-check-target-5ppb9\" (UID: \"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2\") " pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:08.099967 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:08.099957 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:20:08.100030 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:08.099978 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:20:08.100030 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:08.099988 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppnq6 for pod openshift-network-diagnostics/network-check-target-5ppb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:08.100130 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:08.100044 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6 podName:1b1d839b-8804-451e-b28b-ecf0f7a5e5f2 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:16.100025948 +0000 UTC m=+18.233219265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppnq6" (UniqueName: "kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6") pod "network-check-target-5ppb9" (UID: "1b1d839b-8804-451e-b28b-ecf0f7a5e5f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:08.347883 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:08.347707 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:08.347883 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:08.347837 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:08.348886 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:08.348860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:08.349020 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:08.348966 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:09.345584 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:09.345552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:09.346001 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:09.345679 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:10.346297 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:10.345809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:10.346297 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:10.345818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:10.346297 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:10.345938 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:10.346297 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:10.346019 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:11.346022 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:11.345581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:11.346022 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:11.345707 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:12.345452 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:12.345414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:12.345819 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:12.345549 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:12.345819 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:12.345610 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:12.345819 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:12.345711 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:13.345820 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:13.345789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:13.346254 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:13.345896 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:14.345682 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:14.345650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:14.345876 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:14.345691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:14.345876 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:14.345804 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:14.346289 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:14.345942 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:15.345908 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:15.345874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:15.346424 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:15.345982 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:16.065663 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:16.065624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:16.065663 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:16.065678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:16.065902 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:16.065787 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:16.065902 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:16.065811 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:16.065902 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:16.065846 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret podName:4fd5ac79-bd46-4d5e-9c77-9df73609b5b8 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:32.065830543 +0000 UTC m=+34.199023858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret") pod "global-pull-secret-syncer-mw4m6" (UID: "4fd5ac79-bd46-4d5e-9c77-9df73609b5b8") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:16.065902 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:16.065861 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs podName:81be95be-8bfa-45cb-b2d6-30058fbf2e3b nodeName:}" failed. No retries permitted until 2026-07-15 10:20:32.06585418 +0000 UTC m=+34.199047495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs") pod "network-metrics-daemon-cp4gp" (UID: "81be95be-8bfa-45cb-b2d6-30058fbf2e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:16.166463 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:16.166428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnq6\" (UniqueName: \"kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6\") pod \"network-check-target-5ppb9\" (UID: \"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2\") " pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:16.166607 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:16.166594 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:20:16.166651 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:16.166618 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:20:16.166651 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:16.166628 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppnq6 for pod openshift-network-diagnostics/network-check-target-5ppb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:16.166761 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:16.166685 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6 podName:1b1d839b-8804-451e-b28b-ecf0f7a5e5f2 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:32.166666199 +0000 UTC m=+34.299859518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppnq6" (UniqueName: "kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6") pod "network-check-target-5ppb9" (UID: "1b1d839b-8804-451e-b28b-ecf0f7a5e5f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:16.348016 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:16.347990 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:16.348436 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:16.348085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:16.348436 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:16.348110 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:16.348436 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:16.348195 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:17.346008 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:17.345977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:17.346185 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:17.346100 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:18.348489 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:18.348457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:18.348976 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:18.348463 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:18.348976 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:18.348569 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:18.348976 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:18.348655 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:19.345889 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:19.345852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:19.346072 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:19.345954 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:20.347181 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:20.346866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:20.347181 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:20.347004 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:20.347181 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:20.347067 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:20.347181 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:20.347135 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:20.450553 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:20.450205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9dvfc" event={"ID":"3e068402-7f79-4214-b488-5ee985a633ec","Type":"ContainerStarted","Data":"58db170a94704d0d3b2cb0ef7a3132cf673055d46f9ba82f9f819cab89cfe106"} Jul 15 10:20:21.345448 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.345269 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:21.345572 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:21.345536 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:21.476188 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.476169 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:20:21.476919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.476489 2579 generic.go:356] "Generic (PLEG): container finished" podID="492b2069-1e55-477f-b333-19f21a28a3f0" containerID="0bfd21fc9d329f4c7e3e4f58edf2809e5bbc9f535c2dc0da43b863ba9b3c6584" exitCode=1 Jul 15 10:20:21.476919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.476551 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" event={"ID":"492b2069-1e55-477f-b333-19f21a28a3f0","Type":"ContainerStarted","Data":"dcf853378f587d62b9008f18e543bc9f716b1d03f384c660e24cf559ec6ef0e5"} Jul 15 10:20:21.476919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.476616 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" event={"ID":"492b2069-1e55-477f-b333-19f21a28a3f0","Type":"ContainerStarted","Data":"abadaad6f71b02e9edacf7980e29d9defde8a188fb7b429bb7aa2954170e9e42"} Jul 15 10:20:21.476919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.476653 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" event={"ID":"492b2069-1e55-477f-b333-19f21a28a3f0","Type":"ContainerStarted","Data":"62e494bed462f57ba2c734f49582ec918a71f074e8df7bb875070a09ec30004d"} Jul 15 10:20:21.476919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.476682 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" event={"ID":"492b2069-1e55-477f-b333-19f21a28a3f0","Type":"ContainerStarted","Data":"e3a8f854b5f1d4f024694555cafaa202671fd2df6f535950b96c776b057d7456"} Jul 15 10:20:21.476919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.476718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" event={"ID":"492b2069-1e55-477f-b333-19f21a28a3f0","Type":"ContainerDied","Data":"0bfd21fc9d329f4c7e3e4f58edf2809e5bbc9f535c2dc0da43b863ba9b3c6584"} Jul 15 10:20:21.476919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.476788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" event={"ID":"492b2069-1e55-477f-b333-19f21a28a3f0","Type":"ContainerStarted","Data":"6736856ae0769467b807101dfaf97c9f7a690a06a6a58c5fdc6e20a7fb1d90e1"} Jul 15 10:20:21.477966 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.477948 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" event={"ID":"15dab779-ba59-448b-9a1a-6e7e28e5dec9","Type":"ContainerStarted","Data":"4ae303ea5fb9ac288fc5b75263f65a6afe09e5293457ba3a40dc1df04e581ca2"} Jul 15 10:20:21.479381 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.479360 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" event={"ID":"2fd275ac-6422-49df-a98e-ae1756a0e2c8","Type":"ContainerStarted","Data":"3c117c7a834bf10e93b192478bbea7fff7bd5b87ce580f5745a4d0981d78c529"} Jul 15 10:20:21.480715 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.480691 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zn4vg" event={"ID":"cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2","Type":"ContainerStarted","Data":"9442e02852d4745815a4eb6fd20b62bdfbf055548baaae19d7105bc0e0d2dfbc"} Jul 15 10:20:21.482276 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.482249 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-758tc" event={"ID":"a2023908-01cc-4632-86d3-3d4f3b18e4f2","Type":"ContainerStarted","Data":"6eb8c2489f6e5f9afa55dd4cba412ace8e87982fc768e742abd69aee83943fd2"} Jul 15 10:20:21.483635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.483615 2579 generic.go:356] "Generic (PLEG): container finished" podID="2da181a5-da16-4f42-b591-6aef7ed637bc" containerID="412336cdd037ddbfce6e7fe9f1cb8d3b1531a97a90bb44c236cc1c9066037ce1" exitCode=0 Jul 15 10:20:21.483725 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.483697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" event={"ID":"2da181a5-da16-4f42-b591-6aef7ed637bc","Type":"ContainerDied","Data":"412336cdd037ddbfce6e7fe9f1cb8d3b1531a97a90bb44c236cc1c9066037ce1"} Jul 15 10:20:21.485031 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.484999 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q95hm" event={"ID":"e0471e5a-45bf-4183-b286-8904049cb8ce","Type":"ContainerStarted","Data":"57d5c2cd871f437339a99eb9711c00968f02353d337cdf5c9f523137735d22d7"} Jul 15 10:20:21.497952 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.497911 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9dvfc" podStartSLOduration=12.556649991 podStartE2EDuration="23.497898861s" totalImagesPullingTime="10.94124887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:20.484043244 +0000 UTC m=+22.617236607" watchObservedRunningTime="2026-07-15 10:20:21.497898861 +0000 UTC m=+23.631092204" Jul 15 10:20:21.527753 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.527700 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ssrgb" podStartSLOduration=4.445182913 podStartE2EDuration="23.52768822s" totalImagesPullingTime="19.082505307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:21.498227142 +0000 UTC m=+23.631420476" watchObservedRunningTime="2026-07-15 10:20:21.52768822 +0000 UTC m=+23.660881558" Jul 15 10:20:21.545735 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.545695 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q95hm" podStartSLOduration=4.474111187 podStartE2EDuration="23.545686584s" totalImagesPullingTime="19.071575397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:21.545359225 +0000 UTC m=+23.678552563" watchObservedRunningTime="2026-07-15 10:20:21.545686584 +0000 UTC m=+23.678879921" Jul 15 10:20:21.565956 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.565919 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-758tc" podStartSLOduration=4.428097871 podStartE2EDuration="23.565908147s" totalImagesPullingTime="19.137810276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:21.565599104 +0000 UTC m=+23.698792439" watchObservedRunningTime="2026-07-15 10:20:21.565908147 +0000 UTC m=+23.699101485" Jul 15 10:20:21.582664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:21.582626 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zn4vg" podStartSLOduration=4.520850995 podStartE2EDuration="23.582615574s" totalImagesPullingTime="19.061764579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:21.582287634 +0000 UTC m=+23.715480965" watchObservedRunningTime="2026-07-15 10:20:21.582615574 +0000 UTC m=+23.715808962" Jul 15 10:20:22.090430 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.090408 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 10:20:22.275954 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.275775 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T10:20:22.090426379Z","UUID":"fad4a415-763f-43c6-901f-c602fdb68324","Handler":null,"Name":"","Endpoint":""} Jul 15 10:20:22.277413 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.277390 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 10:20:22.277578 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.277432 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 10:20:22.317257 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.317228 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:22.317790 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.317771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:22.345760 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.345716 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:22.345900 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:22.345867 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:22.345967 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.345932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:22.346017 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:22.345996 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:22.490035 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.490006 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" event={"ID":"2fd275ac-6422-49df-a98e-ae1756a0e2c8","Type":"ContainerStarted","Data":"a6f3a742377c694b039fbd9b908029bff302f7f6194e1054c3e8318e28b95469"} Jul 15 10:20:22.491670 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.491641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-97k4j" event={"ID":"79f325cb-b2b5-4788-9fc6-83b05c671817","Type":"ContainerStarted","Data":"590595cd0e520f4a26e28df30d60b29022d73212f550511fe566f42227881aea"} Jul 15 10:20:22.492548 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.492516 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:22.492939 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.492921 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9dvfc" Jul 15 10:20:22.508402 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:22.508361 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-97k4j" podStartSLOduration=5.428252626 podStartE2EDuration="24.508345844s" totalImagesPullingTime="19.080093218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:22.508268992 +0000 UTC m=+24.641462331" watchObservedRunningTime="2026-07-15 10:20:22.508345844 +0000 UTC m=+24.641539205" Jul 15 10:20:23.345408 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:23.345368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:23.345567 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:23.345493 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:23.496891 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:23.496869 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:20:23.497301 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:23.497272 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" event={"ID":"492b2069-1e55-477f-b333-19f21a28a3f0","Type":"ContainerStarted","Data":"e7408798b5ce19b6577ddcb85956358df994cc0d29ac48842499de5be1fb8e3d"} Jul 15 10:20:24.346100 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:24.345918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:24.346294 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:24.346005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:24.346294 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:24.346198 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:24.346415 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:24.346299 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:24.504787 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:24.504751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" event={"ID":"2fd275ac-6422-49df-a98e-ae1756a0e2c8","Type":"ContainerStarted","Data":"e9a978e788d996f79a742c290d3ed1174cb66f793cd71883e4832ba093a3e86e"} Jul 15 10:20:24.525891 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:24.525837 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqgp2" podStartSLOduration=4.7793201199999995 podStartE2EDuration="26.525823256s" totalImagesPullingTime="21.746503136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:24.525169526 +0000 UTC m=+26.658362876" watchObservedRunningTime="2026-07-15 10:20:24.525823256 +0000 UTC m=+26.659016593" Jul 15 10:20:25.345074 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:25.345045 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:25.345221 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:25.345140 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:26.346156 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:26.345993 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:26.347045 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:26.345993 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:26.347045 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:26.346257 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:26.347045 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:26.346294 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:26.513201 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:26.513174 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:20:26.513609 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:26.513583 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" event={"ID":"492b2069-1e55-477f-b333-19f21a28a3f0","Type":"ContainerStarted","Data":"225a97456c20c966c569945d41f9fb42cc0a328d965018e6562c542de5a902b0"} Jul 15 10:20:26.513867 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:26.513840 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:26.513937 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:26.513910 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:26.514060 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:26.514036 2579 scope.go:117] "RemoveContainer" containerID="0bfd21fc9d329f4c7e3e4f58edf2809e5bbc9f535c2dc0da43b863ba9b3c6584" Jul 15 10:20:26.515926 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:26.515899 2579 generic.go:356] "Generic (PLEG): container finished" podID="2da181a5-da16-4f42-b591-6aef7ed637bc" containerID="f15e1cbd2444646d59e46d58a7d4ead7530e1de37cc354c3bf0bcdbb649503e4" exitCode=0 Jul 15 10:20:26.516042 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:26.515947 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" event={"ID":"2da181a5-da16-4f42-b591-6aef7ed637bc","Type":"ContainerDied","Data":"f15e1cbd2444646d59e46d58a7d4ead7530e1de37cc354c3bf0bcdbb649503e4"} Jul 15 10:20:26.533525 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:26.533504 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:27.345718 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.345692 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:27.345948 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:27.345801 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:27.522833 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.522808 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:20:27.523228 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.523141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" event={"ID":"492b2069-1e55-477f-b333-19f21a28a3f0","Type":"ContainerStarted","Data":"3800224915d264d377b76f3b5ed6e0404755160e981f92ef739e76144eb0a311"} Jul 15 10:20:27.523334 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.523318 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:27.540945 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.540915 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:27.552544 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.552505 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" podStartSLOduration=10.417930723 podStartE2EDuration="29.552492248s" totalImagesPullingTime="19.134561525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:27.551159104 +0000 UTC m=+29.684352443" watchObservedRunningTime="2026-07-15 10:20:27.552492248 +0000 UTC m=+29.685685585" Jul 15 10:20:27.764062 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.763995 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5ppb9"] Jul 15 10:20:27.764199 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.764075 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:27.764199 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:27.764147 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:27.767927 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.767902 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cp4gp"] Jul 15 10:20:27.768036 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.768014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:27.768145 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:27.768125 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:27.768589 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.768570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mw4m6"] Jul 15 10:20:27.768677 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:27.768666 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:27.768822 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:27.768799 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:29.346092 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:29.345870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:29.346420 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:29.345891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:29.346420 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:29.346214 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:29.346420 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:29.345892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:29.346420 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:29.346271 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:29.346420 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:29.346341 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:29.529622 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:29.529595 2579 generic.go:356] "Generic (PLEG): container finished" podID="2da181a5-da16-4f42-b591-6aef7ed637bc" containerID="6fc47b09b40f5c98466f60ebc7dc86abaa5b1c3f6435f43a2f36170866c446ff" exitCode=0 Jul 15 10:20:29.529821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:29.529626 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" event={"ID":"2da181a5-da16-4f42-b591-6aef7ed637bc","Type":"ContainerDied","Data":"6fc47b09b40f5c98466f60ebc7dc86abaa5b1c3f6435f43a2f36170866c446ff"} Jul 15 10:20:31.345520 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:31.345492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:31.346029 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:31.345581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:31.346029 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:31.345586 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mw4m6" podUID="4fd5ac79-bd46-4d5e-9c77-9df73609b5b8" Jul 15 10:20:31.346029 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:31.345606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:31.346029 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:31.345678 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cp4gp" podUID="81be95be-8bfa-45cb-b2d6-30058fbf2e3b" Jul 15 10:20:31.346029 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:31.345728 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ppb9" podUID="1b1d839b-8804-451e-b28b-ecf0f7a5e5f2" Jul 15 10:20:31.535404 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:31.535379 2579 generic.go:356] "Generic (PLEG): container finished" podID="2da181a5-da16-4f42-b591-6aef7ed637bc" containerID="3c5f92abebaba66247e2cd730d555a489875ee40b21f69afda9be95e35a5d978" exitCode=0 Jul 15 10:20:31.535529 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:31.535412 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" event={"ID":"2da181a5-da16-4f42-b591-6aef7ed637bc","Type":"ContainerDied","Data":"3c5f92abebaba66247e2cd730d555a489875ee40b21f69afda9be95e35a5d978"} Jul 15 10:20:32.094161 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:32.094126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:32.094322 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:32.094186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:32.094322 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:32.094287 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:32.094412 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:32.094347 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret podName:4fd5ac79-bd46-4d5e-9c77-9df73609b5b8 nodeName:}" failed. No retries permitted until 2026-07-15 10:21:04.094326755 +0000 UTC m=+66.227520071 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret") pod "global-pull-secret-syncer-mw4m6" (UID: "4fd5ac79-bd46-4d5e-9c77-9df73609b5b8") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:20:32.094412 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:32.094371 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:32.094412 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:32.094410 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs podName:81be95be-8bfa-45cb-b2d6-30058fbf2e3b nodeName:}" failed. No retries permitted until 2026-07-15 10:21:04.094397603 +0000 UTC m=+66.227590920 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs") pod "network-metrics-daemon-cp4gp" (UID: "81be95be-8bfa-45cb-b2d6-30058fbf2e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:20:32.194832 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:32.194801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnq6\" (UniqueName: \"kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6\") pod \"network-check-target-5ppb9\" (UID: \"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2\") " pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:32.195000 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:32.194914 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:20:32.195000 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:32.194928 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:20:32.195000 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:32.194937 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ppnq6 for pod openshift-network-diagnostics/network-check-target-5ppb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:32.195000 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:32.194978 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6 podName:1b1d839b-8804-451e-b28b-ecf0f7a5e5f2 nodeName:}" failed. No retries permitted until 2026-07-15 10:21:04.194966301 +0000 UTC m=+66.328159616 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ppnq6" (UniqueName: "kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6") pod "network-check-target-5ppb9" (UID: "1b1d839b-8804-451e-b28b-ecf0f7a5e5f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:20:33.207201 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.207173 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-247.ec2.internal" event="NodeReady" Jul 15 10:20:33.207716 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.207294 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 10:20:33.246998 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.246974 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85b5bb57fb-wctrw"] Jul 15 10:20:33.248966 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.248945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.252167 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.252136 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 10:20:33.252167 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.252151 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 10:20:33.252347 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.252177 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 10:20:33.254598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.253048 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qn4gb\"" Jul 15 10:20:33.257281 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.257258 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-86hn4"] Jul 15 10:20:33.260071 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.260036 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6snt9"] Jul 15 10:20:33.260220 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.260201 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.261949 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.261895 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 10:20:33.262046 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.261982 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6snt9" Jul 15 10:20:33.262610 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.262591 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 10:20:33.262714 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.262651 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jf9sn\"" Jul 15 10:20:33.262831 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.262814 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 10:20:33.264972 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.264951 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 10:20:33.265067 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.264989 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 10:20:33.265132 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.265114 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85b5bb57fb-wctrw"] Jul 15 10:20:33.265282 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.265269 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vwrs8\"" Jul 15 10:20:33.265375 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.265341 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 10:20:33.268963 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.268942 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-86hn4"] Jul 15 10:20:33.274614 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.274593 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6snt9"] Jul 15 10:20:33.345864 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.345835 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:20:33.346003 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.345863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:20:33.346003 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.345845 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:20:33.348719 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.348447 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fdfwf\"" Jul 15 10:20:33.348719 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.348458 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 10:20:33.348719 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.348453 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 10:20:33.348719 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.348545 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krhcr\"" Jul 15 10:20:33.348990 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.348795 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 10:20:33.349111 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.349094 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 10:20:33.402478 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-bound-sa-token\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.402582 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-registry-certificates\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.402582 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-trusted-ca\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.402582 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-installation-pull-secrets\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.402582 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dwc5\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-kube-api-access-7dwc5\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.402582 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f90c6a02-0588-405c-baf2-2548648e031e-ca-trust-extracted\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.402582 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3decd-de00-4e20-bb88-e5bde87107be-tmp-dir\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.402811 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv2t8\" (UniqueName: \"kubernetes.io/projected/86a3decd-de00-4e20-bb88-e5bde87107be-kube-api-access-gv2t8\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.402811 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ca3ff4f-3931-43fa-ad4d-cf88057f88f3-cert\") pod \"ingress-canary-6snt9\" (UID: \"2ca3ff4f-3931-43fa-ad4d-cf88057f88f3\") " pod="openshift-ingress-canary/ingress-canary-6snt9" Jul 15 10:20:33.402811 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.402811 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-image-registry-private-configuration\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.402944 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.402944 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402885 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gg67\" (UniqueName: \"kubernetes.io/projected/2ca3ff4f-3931-43fa-ad4d-cf88057f88f3-kube-api-access-4gg67\") pod \"ingress-canary-6snt9\" (UID: \"2ca3ff4f-3931-43fa-ad4d-cf88057f88f3\") " pod="openshift-ingress-canary/ingress-canary-6snt9" Jul 15 10:20:33.402944 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.402908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86a3decd-de00-4e20-bb88-e5bde87107be-config-volume\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.504069 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f90c6a02-0588-405c-baf2-2548648e031e-ca-trust-extracted\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.504069 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3decd-de00-4e20-bb88-e5bde87107be-tmp-dir\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.504069 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv2t8\" (UniqueName: \"kubernetes.io/projected/86a3decd-de00-4e20-bb88-e5bde87107be-kube-api-access-gv2t8\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ca3ff4f-3931-43fa-ad4d-cf88057f88f3-cert\") pod \"ingress-canary-6snt9\" (UID: \"2ca3ff4f-3931-43fa-ad4d-cf88057f88f3\") " pod="openshift-ingress-canary/ingress-canary-6snt9" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-image-registry-private-configuration\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gg67\" (UniqueName: \"kubernetes.io/projected/2ca3ff4f-3931-43fa-ad4d-cf88057f88f3-kube-api-access-4gg67\") pod \"ingress-canary-6snt9\" (UID: \"2ca3ff4f-3931-43fa-ad4d-cf88057f88f3\") " pod="openshift-ingress-canary/ingress-canary-6snt9" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86a3decd-de00-4e20-bb88-e5bde87107be-config-volume\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-bound-sa-token\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-registry-certificates\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-trusted-ca\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-installation-pull-secrets\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.504354 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwc5\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-kube-api-access-7dwc5\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.504894 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504445 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f90c6a02-0588-405c-baf2-2548648e031e-ca-trust-extracted\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.505019 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.504996 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3decd-de00-4e20-bb88-e5bde87107be-tmp-dir\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.505124 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.505105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86a3decd-de00-4e20-bb88-e5bde87107be-config-volume\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.505268 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.505249 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-registry-certificates\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.505730 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:33.505518 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 10:20:33.505730 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:33.505535 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b5bb57fb-wctrw: secret "image-registry-tls" not found Jul 15 10:20:33.505730 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:33.505600 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls podName:f90c6a02-0588-405c-baf2-2548648e031e nodeName:}" failed. No retries permitted until 2026-07-15 10:20:34.00558503 +0000 UTC m=+36.138778348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls") pod "image-registry-85b5bb57fb-wctrw" (UID: "f90c6a02-0588-405c-baf2-2548648e031e") : secret "image-registry-tls" not found Jul 15 10:20:33.505730 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:33.505653 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 10:20:33.505730 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:33.505700 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls podName:86a3decd-de00-4e20-bb88-e5bde87107be nodeName:}" failed. No retries permitted until 2026-07-15 10:20:34.005684824 +0000 UTC m=+36.138878153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls") pod "dns-default-86hn4" (UID: "86a3decd-de00-4e20-bb88-e5bde87107be") : secret "dns-default-metrics-tls" not found Jul 15 10:20:33.506265 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.506245 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-trusted-ca\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.509539 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.509517 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-installation-pull-secrets\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.509645 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.509568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ca3ff4f-3931-43fa-ad4d-cf88057f88f3-cert\") pod \"ingress-canary-6snt9\" (UID: \"2ca3ff4f-3931-43fa-ad4d-cf88057f88f3\") " pod="openshift-ingress-canary/ingress-canary-6snt9" Jul 15 10:20:33.509704 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.509678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-image-registry-private-configuration\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.515107 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.515079 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv2t8\" (UniqueName: \"kubernetes.io/projected/86a3decd-de00-4e20-bb88-e5bde87107be-kube-api-access-gv2t8\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:33.515219 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.515201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dwc5\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-kube-api-access-7dwc5\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.515494 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.515458 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-bound-sa-token\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:33.515791 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.515768 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gg67\" (UniqueName: \"kubernetes.io/projected/2ca3ff4f-3931-43fa-ad4d-cf88057f88f3-kube-api-access-4gg67\") pod \"ingress-canary-6snt9\" (UID: \"2ca3ff4f-3931-43fa-ad4d-cf88057f88f3\") " pod="openshift-ingress-canary/ingress-canary-6snt9" Jul 15 10:20:33.581666 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.581641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6snt9" Jul 15 10:20:33.739876 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:33.739651 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6snt9"] Jul 15 10:20:33.745259 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:20:33.745234 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca3ff4f_3931_43fa_ad4d_cf88057f88f3.slice/crio-9694ce70cdc627b25980f113ee40e959507cab9f19d413e05450b4f7c0b96acc WatchSource:0}: Error finding container 9694ce70cdc627b25980f113ee40e959507cab9f19d413e05450b4f7c0b96acc: Status 404 returned error can't find the container with id 9694ce70cdc627b25980f113ee40e959507cab9f19d413e05450b4f7c0b96acc Jul 15 10:20:34.008557 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:34.008524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:34.008777 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:34.008590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:34.008777 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:34.008690 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 10:20:34.008777 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:34.008706 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b5bb57fb-wctrw: secret "image-registry-tls" not found Jul 15 10:20:34.008777 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:34.008723 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 10:20:34.008951 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:34.008799 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls podName:f90c6a02-0588-405c-baf2-2548648e031e nodeName:}" failed. No retries permitted until 2026-07-15 10:20:35.008781337 +0000 UTC m=+37.141974652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls") pod "image-registry-85b5bb57fb-wctrw" (UID: "f90c6a02-0588-405c-baf2-2548648e031e") : secret "image-registry-tls" not found Jul 15 10:20:34.008951 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:34.008815 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls podName:86a3decd-de00-4e20-bb88-e5bde87107be nodeName:}" failed. No retries permitted until 2026-07-15 10:20:35.008807448 +0000 UTC m=+37.142000764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls") pod "dns-default-86hn4" (UID: "86a3decd-de00-4e20-bb88-e5bde87107be") : secret "dns-default-metrics-tls" not found Jul 15 10:20:34.543695 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:34.543650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6snt9" event={"ID":"2ca3ff4f-3931-43fa-ad4d-cf88057f88f3","Type":"ContainerStarted","Data":"9694ce70cdc627b25980f113ee40e959507cab9f19d413e05450b4f7c0b96acc"} Jul 15 10:20:35.017102 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:35.017063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:35.017284 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:35.017137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:35.017284 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:35.017208 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 10:20:35.017284 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:35.017230 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b5bb57fb-wctrw: secret "image-registry-tls" not found Jul 15 10:20:35.017284 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:35.017267 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 10:20:35.017434 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:35.017298 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls podName:f90c6a02-0588-405c-baf2-2548648e031e nodeName:}" failed. No retries permitted until 2026-07-15 10:20:37.017277732 +0000 UTC m=+39.150471050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls") pod "image-registry-85b5bb57fb-wctrw" (UID: "f90c6a02-0588-405c-baf2-2548648e031e") : secret "image-registry-tls" not found Jul 15 10:20:35.017434 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:35.017328 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls podName:86a3decd-de00-4e20-bb88-e5bde87107be nodeName:}" failed. No retries permitted until 2026-07-15 10:20:37.017312446 +0000 UTC m=+39.150505763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls") pod "dns-default-86hn4" (UID: "86a3decd-de00-4e20-bb88-e5bde87107be") : secret "dns-default-metrics-tls" not found Jul 15 10:20:35.371639 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:35.371566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:35.483483 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:35.483447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:36.318706 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:36.318670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:36.552309 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:36.552277 2579 generic.go:356] "Generic (PLEG): container finished" podID="2da181a5-da16-4f42-b591-6aef7ed637bc" containerID="81ebf908746290d0ee63c6c65834f00c09a5235517cec3141a6e17147d714184" exitCode=0 Jul 15 10:20:36.552464 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:36.552328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" event={"ID":"2da181a5-da16-4f42-b591-6aef7ed637bc","Type":"ContainerDied","Data":"81ebf908746290d0ee63c6c65834f00c09a5235517cec3141a6e17147d714184"} Jul 15 10:20:37.034602 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:37.034569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:37.034817 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:37.034637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:37.034817 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:37.034710 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 10:20:37.034817 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:37.034733 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b5bb57fb-wctrw: secret "image-registry-tls" not found Jul 15 10:20:37.034817 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:37.034786 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 10:20:37.034817 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:37.034803 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls podName:f90c6a02-0588-405c-baf2-2548648e031e nodeName:}" failed. No retries permitted until 2026-07-15 10:20:41.034786829 +0000 UTC m=+43.167980150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls") pod "image-registry-85b5bb57fb-wctrw" (UID: "f90c6a02-0588-405c-baf2-2548648e031e") : secret "image-registry-tls" not found Jul 15 10:20:37.035020 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:37.034838 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls podName:86a3decd-de00-4e20-bb88-e5bde87107be nodeName:}" failed. No retries permitted until 2026-07-15 10:20:41.034822611 +0000 UTC m=+43.168015932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls") pod "dns-default-86hn4" (UID: "86a3decd-de00-4e20-bb88-e5bde87107be") : secret "dns-default-metrics-tls" not found Jul 15 10:20:37.557540 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:37.557505 2579 generic.go:356] "Generic (PLEG): container finished" podID="2da181a5-da16-4f42-b591-6aef7ed637bc" containerID="974d9563f1dc4285ad7569bfc41a5cc54d57803193fed64dfc1e01449d48c0b2" exitCode=0 Jul 15 10:20:37.557998 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:37.557582 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" event={"ID":"2da181a5-da16-4f42-b591-6aef7ed637bc","Type":"ContainerDied","Data":"974d9563f1dc4285ad7569bfc41a5cc54d57803193fed64dfc1e01449d48c0b2"} Jul 15 10:20:37.563073 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:37.563043 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6snt9" event={"ID":"2ca3ff4f-3931-43fa-ad4d-cf88057f88f3","Type":"ContainerStarted","Data":"97d23edda1b0e8fdeaa264ee2161e75ee9643436f6d2f34b9bc1929af5936ced"} Jul 15 10:20:37.604614 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:37.604370 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6snt9" podStartSLOduration=0.942113376 podStartE2EDuration="4.604349748s" totalImagesPullingTime="3.662236372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:37.604262234 +0000 UTC m=+39.737455563" watchObservedRunningTime="2026-07-15 10:20:37.604349748 +0000 UTC m=+39.737543087" Jul 15 10:20:38.568807 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:38.568768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" event={"ID":"2da181a5-da16-4f42-b591-6aef7ed637bc","Type":"ContainerStarted","Data":"a7b6e6b79ae7a8469aee54584eee7d2741bef1982fd27c08fbfa88f0fde0c688"} Jul 15 10:20:38.595669 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:38.595619 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xhxtz" podStartSLOduration=10.381520208 podStartE2EDuration="40.595605566s" totalImagesPullingTime="30.214085358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:38.594621498 +0000 UTC m=+40.727814836" watchObservedRunningTime="2026-07-15 10:20:38.595605566 +0000 UTC m=+40.728798905" Jul 15 10:20:38.726601 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:38.726571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:38.828763 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:38.828685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:38.924796 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:38.924767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:39.243960 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:39.243930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:39.344569 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:39.344538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:40.840377 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.840339 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sprhs"] Jul 15 10:20:40.860937 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.860913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:40.876282 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.876264 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sprhs"] Jul 15 10:20:40.876366 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.876354 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:40.879659 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.879640 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 10:20:40.879659 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.879657 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 10:20:40.879822 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.879674 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b6kmk\"" Jul 15 10:20:40.879822 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.879662 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 10:20:40.879822 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.879683 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 10:20:40.927711 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.927692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:40.969929 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.969908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2fcae6d-fd94-42d8-82fd-606fa907ca19-data-volume\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:40.970001 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.969945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:40.970001 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.969969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:40.970070 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.970038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2fcae6d-fd94-42d8-82fd-606fa907ca19-crio-socket\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:40.970106 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:40.970093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p7j4\" (UniqueName: \"kubernetes.io/projected/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-api-access-2p7j4\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.071145 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.071121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2fcae6d-fd94-42d8-82fd-606fa907ca19-crio-socket\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.071227 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.071158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:41.071227 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.071182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7j4\" (UniqueName: \"kubernetes.io/projected/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-api-access-2p7j4\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.071297 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.071231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2fcae6d-fd94-42d8-82fd-606fa907ca19-data-volume\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.071297 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.071269 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 10:20:41.071356 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.071334 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls podName:86a3decd-de00-4e20-bb88-e5bde87107be nodeName:}" failed. No retries permitted until 2026-07-15 10:20:49.071318611 +0000 UTC m=+51.204511927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls") pod "dns-default-86hn4" (UID: "86a3decd-de00-4e20-bb88-e5bde87107be") : secret "dns-default-metrics-tls" not found Jul 15 10:20:41.071356 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.071344 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2fcae6d-fd94-42d8-82fd-606fa907ca19-crio-socket\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.071429 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.071348 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 10:20:41.071429 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.071417 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls podName:f2fcae6d-fd94-42d8-82fd-606fa907ca19 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:41.571402503 +0000 UTC m=+43.704595819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sprhs" (UID: "f2fcae6d-fd94-42d8-82fd-606fa907ca19") : secret "insights-runtime-extractor-tls" not found Jul 15 10:20:41.071429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.071270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.071535 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.071461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.071535 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.071485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:41.071599 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.071554 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 10:20:41.071599 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.071563 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b5bb57fb-wctrw: secret "image-registry-tls" not found Jul 15 10:20:41.071599 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.071570 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 10:20:41.071599 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.071597 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls podName:f90c6a02-0588-405c-baf2-2548648e031e nodeName:}" failed. No retries permitted until 2026-07-15 10:20:49.07158522 +0000 UTC m=+51.204778539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls") pod "image-registry-85b5bb57fb-wctrw" (UID: "f90c6a02-0588-405c-baf2-2548648e031e") : secret "image-registry-tls" not found Jul 15 10:20:41.071734 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.071605 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2fcae6d-fd94-42d8-82fd-606fa907ca19-data-volume\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.071734 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.071622 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm podName:f2fcae6d-fd94-42d8-82fd-606fa907ca19 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:41.571610733 +0000 UTC m=+43.704804049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sprhs" (UID: "f2fcae6d-fd94-42d8-82fd-606fa907ca19") : configmap "kube-rbac-proxy" not found Jul 15 10:20:41.085116 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.085085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7j4\" (UniqueName: \"kubernetes.io/projected/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-api-access-2p7j4\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.131487 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.131465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:41.241514 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.241486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:41.328611 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.328586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:41.432040 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.431988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:41.540373 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.540346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:41.573985 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.573961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.574062 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.573993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:41.574062 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.574052 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 10:20:41.574156 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.574093 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 10:20:41.574156 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.574096 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm podName:f2fcae6d-fd94-42d8-82fd-606fa907ca19 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:42.574081942 +0000 UTC m=+44.707275261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sprhs" (UID: "f2fcae6d-fd94-42d8-82fd-606fa907ca19") : configmap "kube-rbac-proxy" not found Jul 15 10:20:41.574241 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.574181 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls podName:f2fcae6d-fd94-42d8-82fd-606fa907ca19 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:42.574166173 +0000 UTC m=+44.707359488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sprhs" (UID: "f2fcae6d-fd94-42d8-82fd-606fa907ca19") : secret "insights-runtime-extractor-tls" not found Jul 15 10:20:41.641081 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.641057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:41.805789 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.805726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:41.831602 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.831580 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-r2s6b"] Jul 15 10:20:41.853079 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.853060 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-r2s6b"] Jul 15 10:20:41.853361 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.853133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:41.855697 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.855677 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 15 10:20:41.855789 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.855693 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-962vs\"" Jul 15 10:20:41.855789 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.855685 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 15 10:20:41.877163 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.877138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d775f48e-8d09-4df4-818f-29209bcafcac-nginx-conf\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:41.877254 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.877188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:41.877955 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.877937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:41.978381 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.978357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d775f48e-8d09-4df4-818f-29209bcafcac-nginx-conf\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:41.978477 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.978408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:41.978544 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.978492 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 10:20:41.978544 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:41.978536 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert podName:d775f48e-8d09-4df4-818f-29209bcafcac nodeName:}" failed. No retries permitted until 2026-07-15 10:20:42.478524247 +0000 UTC m=+44.611717563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-r2s6b" (UID: "d775f48e-8d09-4df4-818f-29209bcafcac") : secret "networking-console-plugin-cert" not found Jul 15 10:20:41.979015 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.978995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d775f48e-8d09-4df4-818f-29209bcafcac-nginx-conf\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:41.980366 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:41.980348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:42.076492 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:42.076447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:42.176479 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:42.176456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:42.406504 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:42.406484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:42.466339 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:42.466316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:42.483529 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:42.483505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:42.483671 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:42.483655 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 10:20:42.483759 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:42.483724 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert podName:d775f48e-8d09-4df4-818f-29209bcafcac nodeName:}" failed. No retries permitted until 2026-07-15 10:20:43.483706783 +0000 UTC m=+45.616900114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-r2s6b" (UID: "d775f48e-8d09-4df4-818f-29209bcafcac") : secret "networking-console-plugin-cert" not found Jul 15 10:20:42.584759 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:42.584719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:42.584869 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:42.584777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:42.584869 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:42.584847 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 10:20:42.584953 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:42.584892 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls podName:f2fcae6d-fd94-42d8-82fd-606fa907ca19 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:44.584879564 +0000 UTC m=+46.718072880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sprhs" (UID: "f2fcae6d-fd94-42d8-82fd-606fa907ca19") : secret "insights-runtime-extractor-tls" not found Jul 15 10:20:42.584953 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:42.584905 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 10:20:42.584953 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:42.584947 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm podName:f2fcae6d-fd94-42d8-82fd-606fa907ca19 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:44.584935185 +0000 UTC m=+46.718128501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sprhs" (UID: "f2fcae6d-fd94-42d8-82fd-606fa907ca19") : configmap "kube-rbac-proxy" not found Jul 15 10:20:42.815532 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:42.815480 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q95hm_e0471e5a-45bf-4183-b286-8904049cb8ce/dns-node-resolver/0.log" Jul 15 10:20:42.929107 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:42.929080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:43.043736 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.043716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:43.414434 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.414410 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-mwtqm"] Jul 15 10:20:43.433448 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.433426 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-mwtqm"] Jul 15 10:20:43.433552 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.433537 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.438982 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.438965 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 15 10:20:43.439091 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.439022 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 15 10:20:43.439091 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.439064 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 15 10:20:43.440003 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.439957 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 15 10:20:43.440147 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.440126 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-92r2s\"" Jul 15 10:20:43.476297 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.476277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:43.491474 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.491455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:43.491567 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.491485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn72p\" (UniqueName: \"kubernetes.io/projected/a839035d-6021-44de-b1e2-3417046f35a6-kube-api-access-tn72p\") pod \"service-ca-79f68b9f4f-mwtqm\" (UID: \"a839035d-6021-44de-b1e2-3417046f35a6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.491610 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.491575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a839035d-6021-44de-b1e2-3417046f35a6-signing-key\") pod \"service-ca-79f68b9f4f-mwtqm\" (UID: \"a839035d-6021-44de-b1e2-3417046f35a6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.491610 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:43.491593 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 10:20:43.491689 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.491628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a839035d-6021-44de-b1e2-3417046f35a6-signing-cabundle\") pod \"service-ca-79f68b9f4f-mwtqm\" (UID: \"a839035d-6021-44de-b1e2-3417046f35a6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.491689 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:43.491642 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert podName:d775f48e-8d09-4df4-818f-29209bcafcac nodeName:}" failed. No retries permitted until 2026-07-15 10:20:45.491628731 +0000 UTC m=+47.624822047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-r2s6b" (UID: "d775f48e-8d09-4df4-818f-29209bcafcac") : secret "networking-console-plugin-cert" not found Jul 15 10:20:43.534648 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.534628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:43.592731 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.592704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a839035d-6021-44de-b1e2-3417046f35a6-signing-key\") pod \"service-ca-79f68b9f4f-mwtqm\" (UID: \"a839035d-6021-44de-b1e2-3417046f35a6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.592836 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.592777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a839035d-6021-44de-b1e2-3417046f35a6-signing-cabundle\") pod \"service-ca-79f68b9f4f-mwtqm\" (UID: \"a839035d-6021-44de-b1e2-3417046f35a6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.592836 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.592828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn72p\" (UniqueName: \"kubernetes.io/projected/a839035d-6021-44de-b1e2-3417046f35a6-kube-api-access-tn72p\") pod \"service-ca-79f68b9f4f-mwtqm\" (UID: \"a839035d-6021-44de-b1e2-3417046f35a6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.593400 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.593384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a839035d-6021-44de-b1e2-3417046f35a6-signing-cabundle\") pod \"service-ca-79f68b9f4f-mwtqm\" (UID: \"a839035d-6021-44de-b1e2-3417046f35a6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.595291 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.595271 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a839035d-6021-44de-b1e2-3417046f35a6-signing-key\") pod \"service-ca-79f68b9f4f-mwtqm\" (UID: \"a839035d-6021-44de-b1e2-3417046f35a6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.602133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.602112 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn72p\" (UniqueName: \"kubernetes.io/projected/a839035d-6021-44de-b1e2-3417046f35a6-kube-api-access-tn72p\") pod \"service-ca-79f68b9f4f-mwtqm\" (UID: \"a839035d-6021-44de-b1e2-3417046f35a6\") " pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.742831 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.742784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" Jul 15 10:20:43.877841 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:43.877802 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-mwtqm"] Jul 15 10:20:43.881305 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:20:43.881279 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda839035d_6021_44de_b1e2_3417046f35a6.slice/crio-4f39fef46f2769597170dae6490459c639fc558ae0ce52f1b779c107a23a7e1a WatchSource:0}: Error finding container 4f39fef46f2769597170dae6490459c639fc558ae0ce52f1b779c107a23a7e1a: Status 404 returned error can't find the container with id 4f39fef46f2769597170dae6490459c639fc558ae0ce52f1b779c107a23a7e1a Jul 15 10:20:44.015844 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:44.015798 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zn4vg_cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2/node-ca/0.log" Jul 15 10:20:44.416598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:44.416573 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6snt9_2ca3ff4f-3931-43fa-ad4d-cf88057f88f3/serve-healthcheck-canary/0.log" Jul 15 10:20:44.509690 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:44.509660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:44.583562 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:44.583531 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" event={"ID":"a839035d-6021-44de-b1e2-3417046f35a6","Type":"ContainerStarted","Data":"4f39fef46f2769597170dae6490459c639fc558ae0ce52f1b779c107a23a7e1a"} Jul 15 10:20:44.599852 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:44.599829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:44.599982 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:44.599858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:44.600041 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:44.599979 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 10:20:44.600041 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:44.600006 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 10:20:44.600041 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:44.600032 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm podName:f2fcae6d-fd94-42d8-82fd-606fa907ca19 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:48.600016577 +0000 UTC m=+50.733209893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sprhs" (UID: "f2fcae6d-fd94-42d8-82fd-606fa907ca19") : configmap "kube-rbac-proxy" not found Jul 15 10:20:44.600166 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:44.600089 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls podName:f2fcae6d-fd94-42d8-82fd-606fa907ca19 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:48.600076316 +0000 UTC m=+50.733269635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sprhs" (UID: "f2fcae6d-fd94-42d8-82fd-606fa907ca19") : secret "insights-runtime-extractor-tls" not found Jul 15 10:20:45.506263 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:45.506230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:45.506603 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:45.506333 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 10:20:45.506603 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:45.506392 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert podName:d775f48e-8d09-4df4-818f-29209bcafcac nodeName:}" failed. No retries permitted until 2026-07-15 10:20:49.506378197 +0000 UTC m=+51.639571513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-r2s6b" (UID: "d775f48e-8d09-4df4-818f-29209bcafcac") : secret "networking-console-plugin-cert" not found Jul 15 10:20:45.712158 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:45.712131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:45.779031 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:45.778948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:47.128158 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:47.128118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:47.196053 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:47.196030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:47.590677 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:47.590613 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" event={"ID":"a839035d-6021-44de-b1e2-3417046f35a6","Type":"ContainerStarted","Data":"fc63eb1fe1a038442c72d63f7c5da862d1865f5a412e4f37e1ac9a7252b86ca9"} Jul 15 10:20:47.608275 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:47.608222 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-mwtqm" podStartSLOduration=1.916916301 podStartE2EDuration="4.608203803s" totalImagesPullingTime="2.691287502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:20:47.607698922 +0000 UTC m=+49.740892257" watchObservedRunningTime="2026-07-15 10:20:47.608203803 +0000 UTC m=+49.741397142" Jul 15 10:20:47.893018 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:47.892996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:48.355480 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:48.355443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:48.455208 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:48.455173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:48.557425 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:48.557395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:48.630991 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:48.630919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:48.631531 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:48.631505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:48.631708 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:48.631689 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 10:20:48.631808 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:48.631791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2fcae6d-fd94-42d8-82fd-606fa907ca19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:48.631808 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:48.631806 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls podName:f2fcae6d-fd94-42d8-82fd-606fa907ca19 nodeName:}" failed. No retries permitted until 2026-07-15 10:20:56.631785577 +0000 UTC m=+58.764978907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sprhs" (UID: "f2fcae6d-fd94-42d8-82fd-606fa907ca19") : secret "insights-runtime-extractor-tls" not found Jul 15 10:20:48.657845 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:48.657818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:48.760144 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:48.760118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:48.991139 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:48.991037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:49.091037 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:49.091007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:49.135007 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:49.134972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:20:49.135119 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:49.135038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:20:49.135119 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:49.135107 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 10:20:49.135205 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:49.135124 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b5bb57fb-wctrw: secret "image-registry-tls" not found Jul 15 10:20:49.135205 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:49.135188 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls podName:f90c6a02-0588-405c-baf2-2548648e031e nodeName:}" failed. No retries permitted until 2026-07-15 10:21:05.135171102 +0000 UTC m=+67.268364418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls") pod "image-registry-85b5bb57fb-wctrw" (UID: "f90c6a02-0588-405c-baf2-2548648e031e") : secret "image-registry-tls" not found Jul 15 10:20:49.135205 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:49.135196 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 10:20:49.135322 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:49.135252 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls podName:86a3decd-de00-4e20-bb88-e5bde87107be nodeName:}" failed. No retries permitted until 2026-07-15 10:21:05.135233469 +0000 UTC m=+67.268426788 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls") pod "dns-default-86hn4" (UID: "86a3decd-de00-4e20-bb88-e5bde87107be") : secret "dns-default-metrics-tls" not found Jul 15 10:20:49.190995 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:49.190971 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:49.537990 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:49.537965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:49.538324 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:49.538096 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 10:20:49.538324 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:20:49.538145 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert podName:d775f48e-8d09-4df4-818f-29209bcafcac nodeName:}" failed. No retries permitted until 2026-07-15 10:20:57.538133086 +0000 UTC m=+59.671326401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-r2s6b" (UID: "d775f48e-8d09-4df4-818f-29209bcafcac") : secret "networking-console-plugin-cert" not found Jul 15 10:20:50.306980 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:50.306956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:50.472801 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:50.472777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:50.704797 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:50.704772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:50.866042 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:50.866014 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:51.067788 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:51.067699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:52.066588 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:52.066554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:52.166695 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:52.166668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:52.470344 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:52.470310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:52.664855 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:52.664822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:52.862776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:52.862705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:52.966811 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:52.966783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:53.069651 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:53.069625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:54.072829 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:54.072799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:54.466383 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:54.466350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:54.654512 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:54.654468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:54.725524 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:54.725460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:54.861790 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:54.861730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:55.066150 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:55.066082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:55.919208 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:55.919175 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:56.269414 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:56.269295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:56.473515 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:56.473488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:56.667107 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:56.667066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:56.693466 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:56.693431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:56.696001 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:56.695975 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2fcae6d-fd94-42d8-82fd-606fa907ca19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sprhs\" (UID: \"f2fcae6d-fd94-42d8-82fd-606fa907ca19\") " pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:56.785167 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:56.785131 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sprhs" Jul 15 10:20:56.865679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:56.865642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:56.933666 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:56.933582 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sprhs"] Jul 15 10:20:56.938058 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:20:56.938024 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fcae6d_fd94_42d8_82fd_606fa907ca19.slice/crio-8060767b144b08312134991cf7dccc7dc8bde687aca638bf9602af14f5cbdad7 WatchSource:0}: Error finding container 8060767b144b08312134991cf7dccc7dc8bde687aca638bf9602af14f5cbdad7: Status 404 returned error can't find the container with id 8060767b144b08312134991cf7dccc7dc8bde687aca638bf9602af14f5cbdad7 Jul 15 10:20:57.083586 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:57.083548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:57.603158 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:57.603109 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:57.610366 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:57.610333 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d775f48e-8d09-4df4-818f-29209bcafcac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-r2s6b\" (UID: \"d775f48e-8d09-4df4-818f-29209bcafcac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:57.616634 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:57.616587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sprhs" event={"ID":"f2fcae6d-fd94-42d8-82fd-606fa907ca19","Type":"ContainerStarted","Data":"8060767b144b08312134991cf7dccc7dc8bde687aca638bf9602af14f5cbdad7"} Jul 15 10:20:57.762159 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:57.762118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" Jul 15 10:20:57.917380 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:57.917343 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-r2s6b"] Jul 15 10:20:57.921354 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:20:57.921325 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd775f48e_8d09_4df4_818f_29209bcafcac.slice/crio-5c76b00c458028e59f156fa975dba678519c136019857763b6d4a0d4b635947a WatchSource:0}: Error finding container 5c76b00c458028e59f156fa975dba678519c136019857763b6d4a0d4b635947a: Status 404 returned error can't find the container with id 5c76b00c458028e59f156fa975dba678519c136019857763b6d4a0d4b635947a Jul 15 10:20:58.280671 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:58.280583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:58.471727 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:58.471688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:58.620882 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:58.620839 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" event={"ID":"d775f48e-8d09-4df4-818f-29209bcafcac","Type":"ContainerStarted","Data":"5c76b00c458028e59f156fa975dba678519c136019857763b6d4a0d4b635947a"} Jul 15 10:20:58.669180 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:58.669139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:58.839079 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:58.839038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:58.938758 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:58.938629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:58.997810 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:58.997702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:59.066358 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:59.066311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:59.168431 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:59.168391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:59.511732 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:59.511679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:59.546952 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:59.546923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-klwfw" Jul 15 10:20:59.570130 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:59.570106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:59.908169 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:59.908133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:20:59.968551 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:20:59.968511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:00.106222 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:00.106180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:00.165817 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:00.165722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:00.320550 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:00.320515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:00.524509 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:00.524434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:00.724066 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:00.724001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:01.254576 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:01.254547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:01.755014 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:01.754970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:01.821574 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:01.821529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:02.150699 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:02.150666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:02.216004 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:02.215961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:02.634727 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:02.634682 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sprhs" event={"ID":"f2fcae6d-fd94-42d8-82fd-606fa907ca19","Type":"ContainerStarted","Data":"903e141ce56029407cb53ec0417e0559a12af4b6c5f6fd594dd9e4da3f1713ab"} Jul 15 10:21:02.636821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:02.636776 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" event={"ID":"d775f48e-8d09-4df4-818f-29209bcafcac","Type":"ContainerStarted","Data":"a5bc5da55e18a24e93498a452cd4a01863462ab0fe29035057633cd499ea0fa2"} Jul 15 10:21:02.656013 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:02.655952 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-r2s6b" podStartSLOduration=17.64560313 podStartE2EDuration="21.655929729s" totalImagesPullingTime="4.010326599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:02.655404266 +0000 UTC m=+64.788597607" watchObservedRunningTime="2026-07-15 10:21:02.655929729 +0000 UTC m=+64.789123045" Jul 15 10:21:04.160819 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.160778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:21:04.161298 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.160858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:21:04.163601 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.163576 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 10:21:04.164336 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.164312 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 10:21:04.174288 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.174252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81be95be-8bfa-45cb-b2d6-30058fbf2e3b-metrics-certs\") pod \"network-metrics-daemon-cp4gp\" (UID: \"81be95be-8bfa-45cb-b2d6-30058fbf2e3b\") " pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:21:04.174758 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.174719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4fd5ac79-bd46-4d5e-9c77-9df73609b5b8-original-pull-secret\") pod \"global-pull-secret-syncer-mw4m6\" (UID: \"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8\") " pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:21:04.257040 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.256995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mw4m6" Jul 15 10:21:04.261370 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.261329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnq6\" (UniqueName: \"kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6\") pod \"network-check-target-5ppb9\" (UID: \"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2\") " pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:21:04.264304 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.264274 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 10:21:04.272791 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.272763 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krhcr\"" Jul 15 10:21:04.273840 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.273801 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 10:21:04.281606 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.281570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cp4gp" Jul 15 10:21:04.285730 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.285704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppnq6\" (UniqueName: \"kubernetes.io/projected/1b1d839b-8804-451e-b28b-ecf0f7a5e5f2-kube-api-access-ppnq6\") pod \"network-check-target-5ppb9\" (UID: \"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2\") " pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:21:04.566670 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.566574 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fdfwf\"" Jul 15 10:21:04.574832 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:04.574785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:21:05.169792 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.169728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:21:05.170218 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.169815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:21:05.172518 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.172486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86a3decd-de00-4e20-bb88-e5bde87107be-metrics-tls\") pod \"dns-default-86hn4\" (UID: \"86a3decd-de00-4e20-bb88-e5bde87107be\") " pod="openshift-dns/dns-default-86hn4" Jul 15 10:21:05.172689 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.172670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") pod \"image-registry-85b5bb57fb-wctrw\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:21:05.366966 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.366935 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qn4gb\"" Jul 15 10:21:05.375199 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.375069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:21:05.377606 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.377373 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jf9sn\"" Jul 15 10:21:05.385218 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.385188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-86hn4" Jul 15 10:21:05.635336 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.635295 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cp4gp"] Jul 15 10:21:05.641293 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.641264 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5ppb9"] Jul 15 10:21:05.643308 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:21:05.643263 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81be95be_8bfa_45cb_b2d6_30058fbf2e3b.slice/crio-301414286b6dc51988e2f759edc63abe90d5563489648763b0edf82a4ef63cea WatchSource:0}: Error finding container 301414286b6dc51988e2f759edc63abe90d5563489648763b0edf82a4ef63cea: Status 404 returned error can't find the container with id 301414286b6dc51988e2f759edc63abe90d5563489648763b0edf82a4ef63cea Jul 15 10:21:05.649407 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.649376 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sprhs" event={"ID":"f2fcae6d-fd94-42d8-82fd-606fa907ca19","Type":"ContainerStarted","Data":"266c0a4da4cf97bc1eb1a647d1cdffedf931ce37c60f8d31d5f70263e7d9b92d"} Jul 15 10:21:05.650776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.650752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cp4gp" event={"ID":"81be95be-8bfa-45cb-b2d6-30058fbf2e3b","Type":"ContainerStarted","Data":"301414286b6dc51988e2f759edc63abe90d5563489648763b0edf82a4ef63cea"} Jul 15 10:21:05.651926 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:21:05.651899 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1d839b_8804_451e_b28b_ecf0f7a5e5f2.slice/crio-a4d27df334e2f66a6d25fe6dbd88954d96e2d43931f97e69d2d0f565dcc61960 WatchSource:0}: Error finding container a4d27df334e2f66a6d25fe6dbd88954d96e2d43931f97e69d2d0f565dcc61960: Status 404 returned error can't find the container with id a4d27df334e2f66a6d25fe6dbd88954d96e2d43931f97e69d2d0f565dcc61960 Jul 15 10:21:05.879095 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.879063 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mw4m6"] Jul 15 10:21:05.882559 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:21:05.882532 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd5ac79_bd46_4d5e_9c77_9df73609b5b8.slice/crio-51fa8a20791c094d6d38e8410f3cc155e49805508a0822c35bc34d66efedddb5 WatchSource:0}: Error finding container 51fa8a20791c094d6d38e8410f3cc155e49805508a0822c35bc34d66efedddb5: Status 404 returned error can't find the container with id 51fa8a20791c094d6d38e8410f3cc155e49805508a0822c35bc34d66efedddb5 Jul 15 10:21:05.884883 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.884536 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-86hn4"] Jul 15 10:21:05.885588 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:05.885555 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85b5bb57fb-wctrw"] Jul 15 10:21:05.888754 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:21:05.888706 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a3decd_de00_4e20_bb88_e5bde87107be.slice/crio-a2639be653abcbf5a98094199ac5a0c8c2d708b5f4f06828cbea5b7813002211 WatchSource:0}: Error finding container a2639be653abcbf5a98094199ac5a0c8c2d708b5f4f06828cbea5b7813002211: Status 404 returned error can't find the container with id a2639be653abcbf5a98094199ac5a0c8c2d708b5f4f06828cbea5b7813002211 Jul 15 10:21:05.891290 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:21:05.891270 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90c6a02_0588_405c_baf2_2548648e031e.slice/crio-e3732b40f6202ab5c564a639d7185ddd73c783d2abf66e2b161447b127e9dc1b WatchSource:0}: Error finding container e3732b40f6202ab5c564a639d7185ddd73c783d2abf66e2b161447b127e9dc1b: Status 404 returned error can't find the container with id e3732b40f6202ab5c564a639d7185ddd73c783d2abf66e2b161447b127e9dc1b Jul 15 10:21:06.658497 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.658317 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" event={"ID":"f90c6a02-0588-405c-baf2-2548648e031e","Type":"ContainerStarted","Data":"410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90"} Jul 15 10:21:06.658497 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.658388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" event={"ID":"f90c6a02-0588-405c-baf2-2548648e031e","Type":"ContainerStarted","Data":"e3732b40f6202ab5c564a639d7185ddd73c783d2abf66e2b161447b127e9dc1b"} Jul 15 10:21:06.659040 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.658545 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:21:06.660360 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.660314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mw4m6" event={"ID":"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8","Type":"ContainerStarted","Data":"51fa8a20791c094d6d38e8410f3cc155e49805508a0822c35bc34d66efedddb5"} Jul 15 10:21:06.662675 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.662641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5ppb9" event={"ID":"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2","Type":"ContainerStarted","Data":"a4d27df334e2f66a6d25fe6dbd88954d96e2d43931f97e69d2d0f565dcc61960"} Jul 15 10:21:06.664144 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.664114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86hn4" event={"ID":"86a3decd-de00-4e20-bb88-e5bde87107be","Type":"ContainerStarted","Data":"a2639be653abcbf5a98094199ac5a0c8c2d708b5f4f06828cbea5b7813002211"} Jul 15 10:21:06.688119 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.687883 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" podStartSLOduration=52.68786019 podStartE2EDuration="52.68786019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:06.686724406 +0000 UTC m=+68.819917746" watchObservedRunningTime="2026-07-15 10:21:06.68786019 +0000 UTC m=+68.821053532" Jul 15 10:21:06.798761 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.797636 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m"] Jul 15 10:21:06.820833 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.818977 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m"] Jul 15 10:21:06.820833 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.819118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.823635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.823008 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 10:21:06.823635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.823309 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 10:21:06.823635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.823440 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 10:21:06.823635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.823548 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 10:21:06.823635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.823641 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 10:21:06.823635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.823653 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 10:21:06.824100 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.823318 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 10:21:06.880293 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.880204 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85b5bb57fb-wctrw"] Jul 15 10:21:06.886336 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.886009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-hub\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.886336 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.886078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ptn\" (UniqueName: \"kubernetes.io/projected/3d77562b-4bf3-491e-8ffd-94a12abfbb88-kube-api-access-f5ptn\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.886336 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.886116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.886336 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.886147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d77562b-4bf3-491e-8ffd-94a12abfbb88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.886336 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.886177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-ca\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.886336 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.886207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.912643 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.912517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:06.991052 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.988200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-ca\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.991052 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.988266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.991052 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.988327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-hub\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.991052 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.988382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ptn\" (UniqueName: \"kubernetes.io/projected/3d77562b-4bf3-491e-8ffd-94a12abfbb88-kube-api-access-f5ptn\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.991052 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.988426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.991052 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.988456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d77562b-4bf3-491e-8ffd-94a12abfbb88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.991052 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.989297 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d77562b-4bf3-491e-8ffd-94a12abfbb88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:06.998578 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:06.998471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-ca\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:07.006160 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.005567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-hub\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:07.006160 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.006089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:07.006976 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.006904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:07.008047 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.008019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d77562b-4bf3-491e-8ffd-94a12abfbb88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:07.014728 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.014677 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ptn\" (UniqueName: \"kubernetes.io/projected/3d77562b-4bf3-491e-8ffd-94a12abfbb88-kube-api-access-f5ptn\") pod \"cluster-proxy-proxy-agent-7845cbb495-m6n5m\" (UID: \"3d77562b-4bf3-491e-8ffd-94a12abfbb88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:07.073007 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.072904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:07.163133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.162731 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" Jul 15 10:21:07.171963 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.171923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:07.275649 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.275614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:07.362501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.361840 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m"] Jul 15 10:21:07.471344 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:21:07.471250 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d77562b_4bf3_491e_8ffd_94a12abfbb88.slice/crio-e2de1d998cf084aebd820e01196b2009f96f74cf73fb38c866cca2b623a92d27 WatchSource:0}: Error finding container e2de1d998cf084aebd820e01196b2009f96f74cf73fb38c866cca2b623a92d27: Status 404 returned error can't find the container with id e2de1d998cf084aebd820e01196b2009f96f74cf73fb38c866cca2b623a92d27 Jul 15 10:21:07.672863 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:07.672803 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" event={"ID":"3d77562b-4bf3-491e-8ffd-94a12abfbb88","Type":"ContainerStarted","Data":"e2de1d998cf084aebd820e01196b2009f96f74cf73fb38c866cca2b623a92d27"} Jul 15 10:21:11.449925 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:11.449831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:11.551707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:11.551650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:11.635888 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:11.635852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:11.700901 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:11.700792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:12.133364 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:12.133303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:14.560025 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:14.559994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:14.620146 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:14.620111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:15.704383 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.704343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sprhs" event={"ID":"f2fcae6d-fd94-42d8-82fd-606fa907ca19","Type":"ContainerStarted","Data":"ef761c17ebf00ad675ba4dfea3a5e08f6eab89e9dfb2b72961e6603ce8559264"} Jul 15 10:21:15.706365 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.706336 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86hn4" event={"ID":"86a3decd-de00-4e20-bb88-e5bde87107be","Type":"ContainerStarted","Data":"b199a0de0954d94284c1f245d4e1ecaef71e8875041839baea3dd36682635a3e"} Jul 15 10:21:15.706510 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.706392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86hn4" event={"ID":"86a3decd-de00-4e20-bb88-e5bde87107be","Type":"ContainerStarted","Data":"b4533e7ae93b85249254678d921f9a947bc0af72714201f514b33f0fbf51975c"} Jul 15 10:21:15.706510 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.706441 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-86hn4" Jul 15 10:21:15.708326 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.708290 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cp4gp" event={"ID":"81be95be-8bfa-45cb-b2d6-30058fbf2e3b","Type":"ContainerStarted","Data":"d3b9d87b1d35a9fa6fbd3905e6aa63d7638b345f63c9c6f228382791d6e996ca"} Jul 15 10:21:15.708476 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.708348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cp4gp" event={"ID":"81be95be-8bfa-45cb-b2d6-30058fbf2e3b","Type":"ContainerStarted","Data":"8369d729c934817e59dcbe2954d4cb5ffc922c557644093d59bfc27d6e387dd0"} Jul 15 10:21:15.710446 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.710418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" event={"ID":"3d77562b-4bf3-491e-8ffd-94a12abfbb88","Type":"ContainerStarted","Data":"f0d0aa010060eb636b2cb754e3543bd81a3ff0f70d3062538f2ee1f02250be18"} Jul 15 10:21:15.710607 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.710458 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" event={"ID":"3d77562b-4bf3-491e-8ffd-94a12abfbb88","Type":"ContainerStarted","Data":"7a495568c1caf3e46c72d1bdfc0f33bdf11b14a79716e5df6756114fa77e3bcc"} Jul 15 10:21:15.710607 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.710475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" event={"ID":"3d77562b-4bf3-491e-8ffd-94a12abfbb88","Type":"ContainerStarted","Data":"f248469cca35e330ebf78ca36de8a5e439522b944c1159df2868327442529872"} Jul 15 10:21:15.711677 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.711644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mw4m6" event={"ID":"4fd5ac79-bd46-4d5e-9c77-9df73609b5b8","Type":"ContainerStarted","Data":"460c23e46384a42116f62665bb84fabbc7013b209f6f022864a773ea5c8d0537"} Jul 15 10:21:15.713043 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.713016 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5ppb9" event={"ID":"1b1d839b-8804-451e-b28b-ecf0f7a5e5f2","Type":"ContainerStarted","Data":"951fe9df71b30e58b1c2e0fe19774596a27b69a6d270dbd90d3d5a16c451d50c"} Jul 15 10:21:15.713181 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.713167 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:21:15.724988 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.724933 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sprhs" podStartSLOduration=18.077494953 podStartE2EDuration="35.724917576s" totalImagesPullingTime="17.647422623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:20:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:15.723780182 +0000 UTC m=+77.856973520" watchObservedRunningTime="2026-07-15 10:21:15.724917576 +0000 UTC m=+77.858110914" Jul 15 10:21:15.741319 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.741267 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5ppb9" podStartSLOduration=68.614732268 podStartE2EDuration="1m17.741242406s" totalImagesPullingTime="9.126510138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:15.740287107 +0000 UTC m=+77.873480446" watchObservedRunningTime="2026-07-15 10:21:15.741242406 +0000 UTC m=+77.874435743" Jul 15 10:21:15.758995 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.758937 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7845cbb495-m6n5m" podStartSLOduration=2.363306863 podStartE2EDuration="9.75891956s" totalImagesPullingTime="7.395612697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:15.757810785 +0000 UTC m=+77.891004123" watchObservedRunningTime="2026-07-15 10:21:15.75891956 +0000 UTC m=+77.892112932" Jul 15 10:21:15.773437 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.773368 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mw4m6" podStartSLOduration=68.820933399 podStartE2EDuration="1m17.773346754s" totalImagesPullingTime="8.952413355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:15.772374646 +0000 UTC m=+77.905567984" watchObservedRunningTime="2026-07-15 10:21:15.773346754 +0000 UTC m=+77.906540093" Jul 15 10:21:15.792452 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.792393 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cp4gp" podStartSLOduration=72.283447917 podStartE2EDuration="1m17.792372105s" totalImagesPullingTime="5.508924188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:15.79180223 +0000 UTC m=+77.924995569" watchObservedRunningTime="2026-07-15 10:21:15.792372105 +0000 UTC m=+77.925565444" Jul 15 10:21:15.809393 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.809320 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-86hn4" podStartSLOduration=33.924034256 podStartE2EDuration="42.809296491s" totalImagesPullingTime="8.885262235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:15.80888576 +0000 UTC m=+77.942079097" watchObservedRunningTime="2026-07-15 10:21:15.809296491 +0000 UTC m=+77.942489831" Jul 15 10:21:15.886893 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.886854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:15.986137 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:15.986055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:16.086713 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:16.086672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:16.209113 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:16.209083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:16.463044 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:16.463009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:16.563978 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:16.563942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:17.118589 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:17.118545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:17.180139 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:17.180077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:18.213102 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:18.213072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:18.297064 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:18.297031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:19.673054 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:19.673018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:19.748114 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:19.748080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:19.812337 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:19.812300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:21.928113 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:21.928080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:23.655957 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:23.655928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:23.754463 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:23.754434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:23.856200 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:23.856162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:24.014658 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.014583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:24.099732 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.099705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:24.109158 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.109135 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-9cqph"] Jul 15 10:21:24.113937 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.113913 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.116304 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.116281 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 15 10:21:24.116415 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.116396 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xptw9\"" Jul 15 10:21:24.117318 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.117296 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 15 10:21:24.117318 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.117316 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 10:21:24.117481 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.117370 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 15 10:21:24.117481 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.117415 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 10:21:24.117623 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.117607 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 10:21:24.125598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.125577 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-9cqph"] Jul 15 10:21:24.135231 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.135209 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-74flj"] Jul 15 10:21:24.138401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.138383 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.141349 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.141331 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 10:21:24.141475 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.141461 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 10:21:24.142452 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.142434 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 10:21:24.142568 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.142553 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w2c94\"" Jul 15 10:21:24.180759 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.180723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:24.220493 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkk8\" (UniqueName: \"kubernetes.io/projected/ceba8a0a-f212-4835-9f43-32e037872cde-kube-api-access-cvkk8\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.220598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnd9r\" (UniqueName: \"kubernetes.io/projected/ba908431-954b-413f-a500-36ea818f03d8-kube-api-access-gnd9r\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.220598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ceba8a0a-f212-4835-9f43-32e037872cde-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.220598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ceba8a0a-f212-4835-9f43-32e037872cde-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.220598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ceba8a0a-f212-4835-9f43-32e037872cde-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.220861 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba908431-954b-413f-a500-36ea818f03d8-root\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.220861 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-wtmp\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.220861 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceba8a0a-f212-4835-9f43-32e037872cde-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.220861 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-tls\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.220861 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.220861 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba908431-954b-413f-a500-36ea818f03d8-metrics-client-ca\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.221066 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba908431-954b-413f-a500-36ea818f03d8-sys\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.221066 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceba8a0a-f212-4835-9f43-32e037872cde-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.221066 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.221066 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.220956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-textfile\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.251806 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.251769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:24.322072 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ceba8a0a-f212-4835-9f43-32e037872cde-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.322072 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba908431-954b-413f-a500-36ea818f03d8-root\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322211 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-wtmp\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322211 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba908431-954b-413f-a500-36ea818f03d8-root\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322211 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceba8a0a-f212-4835-9f43-32e037872cde-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.322211 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-wtmp\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322411 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-tls\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322411 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322411 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba908431-954b-413f-a500-36ea818f03d8-metrics-client-ca\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322411 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba908431-954b-413f-a500-36ea818f03d8-sys\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322411 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceba8a0a-f212-4835-9f43-32e037872cde-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.322641 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ceba8a0a-f212-4835-9f43-32e037872cde-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.322641 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322482 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba908431-954b-413f-a500-36ea818f03d8-sys\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322641 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322827 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-textfile\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322827 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkk8\" (UniqueName: \"kubernetes.io/projected/ceba8a0a-f212-4835-9f43-32e037872cde-kube-api-access-cvkk8\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.322827 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnd9r\" (UniqueName: \"kubernetes.io/projected/ba908431-954b-413f-a500-36ea818f03d8-kube-api-access-gnd9r\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.322827 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ceba8a0a-f212-4835-9f43-32e037872cde-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.322827 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.322796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ceba8a0a-f212-4835-9f43-32e037872cde-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.323073 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.323033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.323211 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.323186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceba8a0a-f212-4835-9f43-32e037872cde-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.323492 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.323472 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-textfile\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.323658 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.323630 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ceba8a0a-f212-4835-9f43-32e037872cde-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.323730 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.323633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba908431-954b-413f-a500-36ea818f03d8-metrics-client-ca\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.325015 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.324986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-tls\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.325368 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.325348 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba908431-954b-413f-a500-36ea818f03d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.325605 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.325585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ceba8a0a-f212-4835-9f43-32e037872cde-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.325909 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.325893 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceba8a0a-f212-4835-9f43-32e037872cde-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.330412 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.330383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:24.338471 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.338447 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnd9r\" (UniqueName: \"kubernetes.io/projected/ba908431-954b-413f-a500-36ea818f03d8-kube-api-access-gnd9r\") pod \"node-exporter-74flj\" (UID: \"ba908431-954b-413f-a500-36ea818f03d8\") " pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.338911 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.338890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkk8\" (UniqueName: \"kubernetes.io/projected/ceba8a0a-f212-4835-9f43-32e037872cde-kube-api-access-cvkk8\") pod \"kube-state-metrics-785778ff74-9cqph\" (UID: \"ceba8a0a-f212-4835-9f43-32e037872cde\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.423382 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.423353 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" Jul 15 10:21:24.449045 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.449019 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74flj" Jul 15 10:21:24.459750 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:21:24.459708 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba908431_954b_413f_a500_36ea818f03d8.slice/crio-1517de596491b019d7f50299beb7ab76873e1b503a3281efd268982025927d17 WatchSource:0}: Error finding container 1517de596491b019d7f50299beb7ab76873e1b503a3281efd268982025927d17: Status 404 returned error can't find the container with id 1517de596491b019d7f50299beb7ab76873e1b503a3281efd268982025927d17 Jul 15 10:21:24.559525 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.559495 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-9cqph"] Jul 15 10:21:24.563266 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:21:24.563233 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceba8a0a_f212_4835_9f43_32e037872cde.slice/crio-8d81091eb2d7591e562de90f978506fd8741d3fe799f599f879757092d91b918 WatchSource:0}: Error finding container 8d81091eb2d7591e562de90f978506fd8741d3fe799f599f879757092d91b918: Status 404 returned error can't find the container with id 8d81091eb2d7591e562de90f978506fd8741d3fe799f599f879757092d91b918 Jul 15 10:21:24.745525 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.745487 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" event={"ID":"ceba8a0a-f212-4835-9f43-32e037872cde","Type":"ContainerStarted","Data":"8d81091eb2d7591e562de90f978506fd8741d3fe799f599f879757092d91b918"} Jul 15 10:21:24.746648 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:24.746623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74flj" event={"ID":"ba908431-954b-413f-a500-36ea818f03d8","Type":"ContainerStarted","Data":"1517de596491b019d7f50299beb7ab76873e1b503a3281efd268982025927d17"} Jul 15 10:21:25.720130 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:25.720064 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-86hn4" Jul 15 10:21:25.752684 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:25.752646 2579 generic.go:356] "Generic (PLEG): container finished" podID="ba908431-954b-413f-a500-36ea818f03d8" containerID="1aa55a1c42cd2f413a3f25bfddd0eece214a4b1c32f77f5222548f73933c5616" exitCode=0 Jul 15 10:21:25.753152 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:25.752695 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74flj" event={"ID":"ba908431-954b-413f-a500-36ea818f03d8","Type":"ContainerDied","Data":"1aa55a1c42cd2f413a3f25bfddd0eece214a4b1c32f77f5222548f73933c5616"} Jul 15 10:21:25.918341 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:25.918309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:26.014779 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.014715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:26.227394 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.227367 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55b98d786b-52mzv"] Jul 15 10:21:26.231322 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.231268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.234707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.233933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 10:21:26.234707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.234157 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cjoiultm2pdnf\"" Jul 15 10:21:26.234707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.234286 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 10:21:26.234707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.234461 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 10:21:26.234707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.234566 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-c4hkl\"" Jul 15 10:21:26.234707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.234594 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 10:21:26.234707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.234643 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 10:21:26.246259 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.246237 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55b98d786b-52mzv"] Jul 15 10:21:26.339588 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.339557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-grpc-tls\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.339714 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.339594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.339714 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.339651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.339714 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.339687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-tls\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.339833 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.339749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9thq\" (UniqueName: \"kubernetes.io/projected/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-kube-api-access-z9thq\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.339833 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.339795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-metrics-client-ca\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.339833 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.339815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.339833 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.339831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.440444 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.440411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.440444 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.440448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-tls\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.440642 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.440476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9thq\" (UniqueName: \"kubernetes.io/projected/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-kube-api-access-z9thq\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.440642 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.440513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-metrics-client-ca\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.440642 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.440564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.440642 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.440594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.440866 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.440661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-grpc-tls\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.440866 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.440696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.441419 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.441310 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-metrics-client-ca\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.443576 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.443545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.444048 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.444026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.444127 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.444048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.444127 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.444071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-grpc-tls\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.444200 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.444134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.444237 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.444202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-secret-thanos-querier-tls\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.449035 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.449013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9thq\" (UniqueName: \"kubernetes.io/projected/66bb1ec0-627c-495a-9211-46eb2c1a8e0d-kube-api-access-z9thq\") pod \"thanos-querier-55b98d786b-52mzv\" (UID: \"66bb1ec0-627c-495a-9211-46eb2c1a8e0d\") " pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.541832 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.541752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:26.673214 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.673188 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55b98d786b-52mzv"] Jul 15 10:21:26.676381 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:21:26.676357 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bb1ec0_627c_495a_9211_46eb2c1a8e0d.slice/crio-0e53373e9f2841266e8c204e7b1d90b97a1d992137076b3e2004c4b4c155b48a WatchSource:0}: Error finding container 0e53373e9f2841266e8c204e7b1d90b97a1d992137076b3e2004c4b4c155b48a: Status 404 returned error can't find the container with id 0e53373e9f2841266e8c204e7b1d90b97a1d992137076b3e2004c4b4c155b48a Jul 15 10:21:26.758113 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.758087 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" event={"ID":"ceba8a0a-f212-4835-9f43-32e037872cde","Type":"ContainerStarted","Data":"e4b1b03776000754480225d626fd5cb2c8fe5e93dc853c5fef701a881a78c4c7"} Jul 15 10:21:26.758429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.758136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" event={"ID":"ceba8a0a-f212-4835-9f43-32e037872cde","Type":"ContainerStarted","Data":"79de7d984d196cec5d4216ad820c2ef737c766f68b98c245f5b84ff3773d6ee4"} Jul 15 10:21:26.758429 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.758169 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" event={"ID":"ceba8a0a-f212-4835-9f43-32e037872cde","Type":"ContainerStarted","Data":"7c37133d569aebcc759337cd766e9c5c77639bf2970a69b2cf9b867327b09122"} Jul 15 10:21:26.759361 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.759333 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" event={"ID":"66bb1ec0-627c-495a-9211-46eb2c1a8e0d","Type":"ContainerStarted","Data":"0e53373e9f2841266e8c204e7b1d90b97a1d992137076b3e2004c4b4c155b48a"} Jul 15 10:21:26.761193 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.761172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74flj" event={"ID":"ba908431-954b-413f-a500-36ea818f03d8","Type":"ContainerStarted","Data":"cb0391fca9923f7623f1092008a5a247978a07f3fa0e837cb41bb8260be4344f"} Jul 15 10:21:26.761277 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.761215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74flj" event={"ID":"ba908431-954b-413f-a500-36ea818f03d8","Type":"ContainerStarted","Data":"518fb1a54397ccdb47a21364cdf45ae78244cf4466e3d6bdc7e65fdb104ca37f"} Jul 15 10:21:26.778635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.778595 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-9cqph" podStartSLOduration=1.316248406 podStartE2EDuration="2.778581903s" totalImagesPullingTime="1.462333497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:26.777781985 +0000 UTC m=+88.910975323" watchObservedRunningTime="2026-07-15 10:21:26.778581903 +0000 UTC m=+88.911775241" Jul 15 10:21:26.798175 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:26.798107 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-74flj" podStartSLOduration=1.775544668 podStartE2EDuration="2.798093254s" totalImagesPullingTime="944.272927ms" totalInitContainerRuntime="78.275659ms" isStatefulPod=true podCreationTimestamp="2026-07-15 10:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:26.797281379 +0000 UTC m=+88.930474717" watchObservedRunningTime="2026-07-15 10:21:26.798093254 +0000 UTC m=+88.931286592" Jul 15 10:21:28.255170 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:28.255127 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:28.360902 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:28.360875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:28.458562 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:28.458527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:28.682865 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:28.682834 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:21:28.896080 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:28.896058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:28.960971 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:28.960942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:29.221178 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:29.221100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:29.776607 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:29.776568 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" event={"ID":"66bb1ec0-627c-495a-9211-46eb2c1a8e0d","Type":"ContainerStarted","Data":"c4f1121d095c06128d2b902a1438e1f7b79ea43abd0c853f16f13a17f4fb7e90"} Jul 15 10:21:29.777028 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:29.776638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" event={"ID":"66bb1ec0-627c-495a-9211-46eb2c1a8e0d","Type":"ContainerStarted","Data":"e0782b2dd7d111fe6c60fc381719b7de4731b6f221123fc8b1039a418b8510cb"} Jul 15 10:21:29.777028 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:29.776667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" event={"ID":"66bb1ec0-627c-495a-9211-46eb2c1a8e0d","Type":"ContainerStarted","Data":"2d5772e6981c92dc10fdb8f91abb5393d009dc653aab731c9386d3647e9efe40"} Jul 15 10:21:30.370051 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:30.370018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:30.662915 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:30.662841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:30.759645 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:30.759617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:30.782551 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:30.782524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" event={"ID":"66bb1ec0-627c-495a-9211-46eb2c1a8e0d","Type":"ContainerStarted","Data":"38ac9f504e06c24bea1d70ae0b76482cf4521aa78da757a6b803574d540e521c"} Jul 15 10:21:30.783000 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:30.782596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" event={"ID":"66bb1ec0-627c-495a-9211-46eb2c1a8e0d","Type":"ContainerStarted","Data":"b3b9a5045c84e92e9377dbcbeaa69f815a72e387d63b4141458ec6d081cd1a76"} Jul 15 10:21:30.783000 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:30.782640 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" event={"ID":"66bb1ec0-627c-495a-9211-46eb2c1a8e0d","Type":"ContainerStarted","Data":"d2daacc15a139dd8b1f3a5e82a256c1e15158d48144ecb849d9da86d37702cd5"} Jul 15 10:21:30.783000 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:30.782717 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:30.813688 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:30.813646 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" podStartSLOduration=1.667783343 podStartE2EDuration="4.813632045s" totalImagesPullingTime="3.145848702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:21:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:21:30.811918199 +0000 UTC m=+92.945111548" watchObservedRunningTime="2026-07-15 10:21:30.813632045 +0000 UTC m=+92.946825383" Jul 15 10:21:32.124632 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:32.124603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:32.221454 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:32.221427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:32.322076 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:32.322045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:32.422695 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:32.422626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:33.704808 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:33.704766 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" podUID="f90c6a02-0588-405c-baf2-2548648e031e" containerName="registry" containerID="cri-o://410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90" gracePeriod=30 Jul 15 10:21:33.941470 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:33.941447 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:21:34.103723 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.103697 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-registry-certificates\") pod \"f90c6a02-0588-405c-baf2-2548648e031e\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " Jul 15 10:21:34.103899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.103735 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f90c6a02-0588-405c-baf2-2548648e031e-ca-trust-extracted\") pod \"f90c6a02-0588-405c-baf2-2548648e031e\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " Jul 15 10:21:34.103899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.103791 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-bound-sa-token\") pod \"f90c6a02-0588-405c-baf2-2548648e031e\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " Jul 15 10:21:34.103899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.103812 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") pod \"f90c6a02-0588-405c-baf2-2548648e031e\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " Jul 15 10:21:34.103899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.103841 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-image-registry-private-configuration\") pod \"f90c6a02-0588-405c-baf2-2548648e031e\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " Jul 15 10:21:34.103899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.103859 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-installation-pull-secrets\") pod \"f90c6a02-0588-405c-baf2-2548648e031e\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " Jul 15 10:21:34.104134 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.103921 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dwc5\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-kube-api-access-7dwc5\") pod \"f90c6a02-0588-405c-baf2-2548648e031e\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " Jul 15 10:21:34.104134 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.104000 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-trusted-ca\") pod \"f90c6a02-0588-405c-baf2-2548648e031e\" (UID: \"f90c6a02-0588-405c-baf2-2548648e031e\") " Jul 15 10:21:34.104562 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.104226 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f90c6a02-0588-405c-baf2-2548648e031e" (UID: "f90c6a02-0588-405c-baf2-2548648e031e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 10:21:34.106916 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.105170 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f90c6a02-0588-405c-baf2-2548648e031e" (UID: "f90c6a02-0588-405c-baf2-2548648e031e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 10:21:34.107729 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.107676 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f90c6a02-0588-405c-baf2-2548648e031e" (UID: "f90c6a02-0588-405c-baf2-2548648e031e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:21:34.111072 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.110988 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-kube-api-access-7dwc5" (OuterVolumeSpecName: "kube-api-access-7dwc5") pod "f90c6a02-0588-405c-baf2-2548648e031e" (UID: "f90c6a02-0588-405c-baf2-2548648e031e"). InnerVolumeSpecName "kube-api-access-7dwc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:21:34.111434 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.111303 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f90c6a02-0588-405c-baf2-2548648e031e" (UID: "f90c6a02-0588-405c-baf2-2548648e031e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 10:21:34.111566 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.111539 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f90c6a02-0588-405c-baf2-2548648e031e" (UID: "f90c6a02-0588-405c-baf2-2548648e031e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 10:21:34.112348 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.112328 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f90c6a02-0588-405c-baf2-2548648e031e" (UID: "f90c6a02-0588-405c-baf2-2548648e031e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:21:34.115246 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.115225 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90c6a02-0588-405c-baf2-2548648e031e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f90c6a02-0588-405c-baf2-2548648e031e" (UID: "f90c6a02-0588-405c-baf2-2548648e031e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:21:34.205597 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.205553 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-bound-sa-token\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:21:34.205597 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.205593 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-registry-tls\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:21:34.205597 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.205606 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-image-registry-private-configuration\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:21:34.205859 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.205617 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f90c6a02-0588-405c-baf2-2548648e031e-installation-pull-secrets\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:21:34.205859 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.205628 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dwc5\" (UniqueName: \"kubernetes.io/projected/f90c6a02-0588-405c-baf2-2548648e031e-kube-api-access-7dwc5\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:21:34.205859 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.205635 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-trusted-ca\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:21:34.205859 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.205643 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f90c6a02-0588-405c-baf2-2548648e031e-registry-certificates\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:21:34.205859 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.205651 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f90c6a02-0588-405c-baf2-2548648e031e-ca-trust-extracted\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:21:34.798640 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.798539 2579 generic.go:356] "Generic (PLEG): container finished" podID="f90c6a02-0588-405c-baf2-2548648e031e" containerID="410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90" exitCode=0 Jul 15 10:21:34.798640 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.798621 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" Jul 15 10:21:34.798640 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.798629 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" event={"ID":"f90c6a02-0588-405c-baf2-2548648e031e","Type":"ContainerDied","Data":"410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90"} Jul 15 10:21:34.799294 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.798692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b5bb57fb-wctrw" event={"ID":"f90c6a02-0588-405c-baf2-2548648e031e","Type":"ContainerDied","Data":"e3732b40f6202ab5c564a639d7185ddd73c783d2abf66e2b161447b127e9dc1b"} Jul 15 10:21:34.799294 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.798716 2579 scope.go:117] "RemoveContainer" containerID="410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90" Jul 15 10:21:34.808908 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.808885 2579 scope.go:117] "RemoveContainer" containerID="410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90" Jul 15 10:21:34.809293 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:21:34.809272 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90\": container with ID starting with 410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90 not found: ID does not exist" containerID="410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90" Jul 15 10:21:34.809353 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.809307 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90"} err="failed to get container status \"410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90\": rpc error: code = NotFound desc = could not find container \"410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90\": container with ID starting with 410b0d3cd9055ca3846fa54ff6c04b0bb6ab7d45994593f15d76d9ca2aff2a90 not found: ID does not exist" Jul 15 10:21:34.818647 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.818608 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85b5bb57fb-wctrw"] Jul 15 10:21:34.823288 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:34.823253 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85b5bb57fb-wctrw"] Jul 15 10:21:36.164927 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:36.164896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:36.265142 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:36.265070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:36.350145 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:36.350117 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90c6a02-0588-405c-baf2-2548648e031e" path="/var/lib/kubelet/pods/f90c6a02-0588-405c-baf2-2548648e031e/volumes" Jul 15 10:21:36.791977 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:36.791948 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55b98d786b-52mzv" Jul 15 10:21:39.551687 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:39.551652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:39.607151 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:39.607118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:45.435153 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:45.435114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:45.499188 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:45.499133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:45.561186 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:45.561153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:46.720087 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:46.720058 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5ppb9" Jul 15 10:21:47.130979 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:47.130950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:21:47.226199 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:21:47.226169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:02.125179 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:02.125143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:02.206930 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:02.206895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:11.123804 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:11.123766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:40.319521 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:40.319487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:40.381169 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:40.381140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:43.604418 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:43.604387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:45.254583 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:45.254549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:45.351843 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:45.351814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:46.228536 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.228503 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 10:22:46.228811 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.228797 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90c6a02-0588-405c-baf2-2548648e031e" containerName="registry" Jul 15 10:22:46.228866 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.228813 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90c6a02-0588-405c-baf2-2548648e031e" containerName="registry" Jul 15 10:22:46.228909 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.228872 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90c6a02-0588-405c-baf2-2548648e031e" containerName="registry" Jul 15 10:22:46.233482 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.233463 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.235954 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.235931 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nkqnm\"" Jul 15 10:22:46.235954 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.235939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 10:22:46.236115 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.235959 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 10:22:46.236232 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.236218 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 10:22:46.236310 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.236282 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 10:22:46.236369 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.236305 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 10:22:46.236369 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.236305 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 10:22:46.236457 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.236300 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 10:22:46.236457 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.236432 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 10:22:46.241116 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.241098 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 10:22:46.245941 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.245917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 10:22:46.305466 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.305776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.305776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-config-volume\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.305776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfsr\" (UniqueName: \"kubernetes.io/projected/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-kube-api-access-wrfsr\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.305776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.305776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-web-config\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.305776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305634 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.305776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-config-out\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.305776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.305776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-tls-assets\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.305776 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.306075 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.306075 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.305800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406224 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-web-config\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406317 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406317 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-config-out\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406317 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406420 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-tls-assets\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406420 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406500 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406500 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406500 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406637 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406637 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-config-volume\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406637 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfsr\" (UniqueName: \"kubernetes.io/projected/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-kube-api-access-wrfsr\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406637 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.406842 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.406694 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.407126 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.407108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.407351 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.407325 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.409726 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.409314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-config-out\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.410552 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.410261 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-config-volume\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.410552 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.410312 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.410552 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.410351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.410552 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.410504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-web-config\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.410734 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.410561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-tls-assets\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.410734 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.410649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.410833 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.410787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.411624 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.411603 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.418084 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.418066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfsr\" (UniqueName: \"kubernetes.io/projected/78ca7ebb-b1b6-4b52-ba61-860ac7c46c74-kube-api-access-wrfsr\") pod \"alertmanager-main-0\" (UID: \"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.543509 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.543455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:22:46.679039 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:46.678899 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 10:22:46.683783 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:22:46.683732 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ca7ebb_b1b6_4b52_ba61_860ac7c46c74.slice/crio-68da8723bf4184f314ddf7aee58409e4d1838cde48aed531fac34fe84b562fd3 WatchSource:0}: Error finding container 68da8723bf4184f314ddf7aee58409e4d1838cde48aed531fac34fe84b562fd3: Status 404 returned error can't find the container with id 68da8723bf4184f314ddf7aee58409e4d1838cde48aed531fac34fe84b562fd3 Jul 15 10:22:47.071163 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:47.071135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74","Type":"ContainerStarted","Data":"68da8723bf4184f314ddf7aee58409e4d1838cde48aed531fac34fe84b562fd3"} Jul 15 10:22:48.076539 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:48.076474 2579 generic.go:356] "Generic (PLEG): container finished" podID="78ca7ebb-b1b6-4b52-ba61-860ac7c46c74" containerID="e3d67e03b48682811d5e08d77a9b05913d9fc418b305929138c65657c3d167db" exitCode=0 Jul 15 10:22:48.076959 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:48.076547 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74","Type":"ContainerDied","Data":"e3d67e03b48682811d5e08d77a9b05913d9fc418b305929138c65657c3d167db"} Jul 15 10:22:50.087635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:50.087549 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74","Type":"ContainerStarted","Data":"1b5617cd619ca1d4e916f553b5ef7fdf3a23ffdf49ef44b824b0d87593ad7ecf"} Jul 15 10:22:50.087635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:50.087614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74","Type":"ContainerStarted","Data":"902d7aa71eb7717f181ebdc9f9cb04e06210b06c4bf529b18de3d3711f5e4ffb"} Jul 15 10:22:50.088059 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:50.087643 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74","Type":"ContainerStarted","Data":"197fd1be64c24afbc4743a0b2932dc76608b45234b113cf37d83e4e7372adf0c"} Jul 15 10:22:50.088059 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:50.087668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74","Type":"ContainerStarted","Data":"1f7a9a8e42b9aa535493c371effe42eadd873fd510c2af5fa1c9fb8ff9261107"} Jul 15 10:22:50.088059 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:50.087690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74","Type":"ContainerStarted","Data":"b72875ad048b4cdcd15ae54cb018f048203768ede16880b8f83944a596fb6ce9"} Jul 15 10:22:50.088059 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:50.087711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78ca7ebb-b1b6-4b52-ba61-860ac7c46c74","Type":"ContainerStarted","Data":"54df1efbe20bc9030e8f76b0332df59edd6ebb40a3ece482d4b6b5817dce9032"} Jul 15 10:22:50.118157 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:50.118103 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.461446639 podStartE2EDuration="4.118084977s" totalImagesPullingTime="2.637113309s" totalInitContainerRuntime="19.525029ms" isStatefulPod=false podCreationTimestamp="2026-07-15 10:22:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:22:50.11662 +0000 UTC m=+172.249813337" watchObservedRunningTime="2026-07-15 10:22:50.118084977 +0000 UTC m=+172.251278332" Jul 15 10:22:52.130591 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:52.130537 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:52.191405 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:52.191378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:55.717384 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:55.717338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:55.816645 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:55.816616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:22:55.916503 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:22:55.916476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:23:02.126111 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:23:02.126073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:23:02.222392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:23:02.222356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:23:02.282368 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:23:02.282332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:23:31.466777 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:23:31.466724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:23:53.594695 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:23:53.594647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:23:53.687154 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:23:53.687119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:15.299581 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:15.299545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:16.125148 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:16.125113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:17.576001 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.575970 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg"] Jul 15 10:24:17.579180 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.579160 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" Jul 15 10:24:17.583772 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.583729 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 15 10:24:17.583772 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.583755 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 15 10:24:17.584138 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.584123 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hn64g\"" Jul 15 10:24:17.600462 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.600440 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg"] Jul 15 10:24:17.652237 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.652217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2lxd\" (UniqueName: \"kubernetes.io/projected/6441f125-5f9b-41b0-a6f6-0b8e6e739a13-kube-api-access-n2lxd\") pod \"cert-manager-operator-controller-manager-55c444bf86-wqhpg\" (UID: \"6441f125-5f9b-41b0-a6f6-0b8e6e739a13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" Jul 15 10:24:17.652338 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.652269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6441f125-5f9b-41b0-a6f6-0b8e6e739a13-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wqhpg\" (UID: \"6441f125-5f9b-41b0-a6f6-0b8e6e739a13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" Jul 15 10:24:17.752770 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.752726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6441f125-5f9b-41b0-a6f6-0b8e6e739a13-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wqhpg\" (UID: \"6441f125-5f9b-41b0-a6f6-0b8e6e739a13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" Jul 15 10:24:17.752862 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.752802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2lxd\" (UniqueName: \"kubernetes.io/projected/6441f125-5f9b-41b0-a6f6-0b8e6e739a13-kube-api-access-n2lxd\") pod \"cert-manager-operator-controller-manager-55c444bf86-wqhpg\" (UID: \"6441f125-5f9b-41b0-a6f6-0b8e6e739a13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" Jul 15 10:24:17.753087 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.753069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6441f125-5f9b-41b0-a6f6-0b8e6e739a13-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wqhpg\" (UID: \"6441f125-5f9b-41b0-a6f6-0b8e6e739a13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" Jul 15 10:24:17.763304 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.763270 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2lxd\" (UniqueName: \"kubernetes.io/projected/6441f125-5f9b-41b0-a6f6-0b8e6e739a13-kube-api-access-n2lxd\") pod \"cert-manager-operator-controller-manager-55c444bf86-wqhpg\" (UID: \"6441f125-5f9b-41b0-a6f6-0b8e6e739a13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" Jul 15 10:24:17.888960 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:17.888935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" Jul 15 10:24:18.028804 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:18.028765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg"] Jul 15 10:24:18.032219 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:24:18.032194 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6441f125_5f9b_41b0_a6f6_0b8e6e739a13.slice/crio-ef7276fe4d14fd037e962d0d4c1b1f118d6675555e85978000bed2e842983951 WatchSource:0}: Error finding container ef7276fe4d14fd037e962d0d4c1b1f118d6675555e85978000bed2e842983951: Status 404 returned error can't find the container with id ef7276fe4d14fd037e962d0d4c1b1f118d6675555e85978000bed2e842983951 Jul 15 10:24:18.409097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:18.409065 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" event={"ID":"6441f125-5f9b-41b0-a6f6-0b8e6e739a13","Type":"ContainerStarted","Data":"ef7276fe4d14fd037e962d0d4c1b1f118d6675555e85978000bed2e842983951"} Jul 15 10:24:19.119384 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:19.119345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:19.222491 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:19.222455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:21.426721 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:21.426671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" event={"ID":"6441f125-5f9b-41b0-a6f6-0b8e6e739a13","Type":"ContainerStarted","Data":"19bee43ad39e749abd7be1df3eaa7052543a66068891890ffd4c7957015048d3"} Jul 15 10:24:21.459474 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:21.459415 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wqhpg" podStartSLOduration=1.902959335 podStartE2EDuration="4.459394192s" totalImagesPullingTime="2.556434857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:24:21.457925547 +0000 UTC m=+263.591118886" watchObservedRunningTime="2026-07-15 10:24:21.459394192 +0000 UTC m=+263.592587531" Jul 15 10:24:21.517494 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:21.517464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:21.609108 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:21.609079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:21.942176 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:21.942151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:22.036488 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:22.036466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:23.360586 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:23.360554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:23.949390 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:23.949362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:26.347418 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:26.347384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:26.426926 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:26.426902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:29.541457 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.541427 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bgpl4"] Jul 15 10:24:29.544729 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.544711 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" Jul 15 10:24:29.549440 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.549422 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 10:24:29.550328 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.550308 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rgdhr\"" Jul 15 10:24:29.550444 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.550308 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 10:24:29.572614 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.572594 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bgpl4"] Jul 15 10:24:29.639043 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.639017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhffj\" (UniqueName: \"kubernetes.io/projected/e68f98d8-e417-41cf-8c26-4dd54af6229a-kube-api-access-hhffj\") pod \"cert-manager-webhook-6486ffd997-bgpl4\" (UID: \"e68f98d8-e417-41cf-8c26-4dd54af6229a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" Jul 15 10:24:29.639138 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.639045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e68f98d8-e417-41cf-8c26-4dd54af6229a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bgpl4\" (UID: \"e68f98d8-e417-41cf-8c26-4dd54af6229a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" Jul 15 10:24:29.740316 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.740292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhffj\" (UniqueName: \"kubernetes.io/projected/e68f98d8-e417-41cf-8c26-4dd54af6229a-kube-api-access-hhffj\") pod \"cert-manager-webhook-6486ffd997-bgpl4\" (UID: \"e68f98d8-e417-41cf-8c26-4dd54af6229a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" Jul 15 10:24:29.740403 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.740317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e68f98d8-e417-41cf-8c26-4dd54af6229a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bgpl4\" (UID: \"e68f98d8-e417-41cf-8c26-4dd54af6229a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" Jul 15 10:24:29.759220 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.759193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhffj\" (UniqueName: \"kubernetes.io/projected/e68f98d8-e417-41cf-8c26-4dd54af6229a-kube-api-access-hhffj\") pod \"cert-manager-webhook-6486ffd997-bgpl4\" (UID: \"e68f98d8-e417-41cf-8c26-4dd54af6229a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" Jul 15 10:24:29.764160 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.764141 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e68f98d8-e417-41cf-8c26-4dd54af6229a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bgpl4\" (UID: \"e68f98d8-e417-41cf-8c26-4dd54af6229a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" Jul 15 10:24:29.854369 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.854345 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" Jul 15 10:24:29.994945 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:29.994912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bgpl4"] Jul 15 10:24:29.999002 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:24:29.998973 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode68f98d8_e417_41cf_8c26_4dd54af6229a.slice/crio-7bd07fce406c5a7d37fdf28ac2d77fef7e1ca18cb00cf74c9cefbfb2e36fba26 WatchSource:0}: Error finding container 7bd07fce406c5a7d37fdf28ac2d77fef7e1ca18cb00cf74c9cefbfb2e36fba26: Status 404 returned error can't find the container with id 7bd07fce406c5a7d37fdf28ac2d77fef7e1ca18cb00cf74c9cefbfb2e36fba26 Jul 15 10:24:30.461133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:30.461096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" event={"ID":"e68f98d8-e417-41cf-8c26-4dd54af6229a","Type":"ContainerStarted","Data":"7bd07fce406c5a7d37fdf28ac2d77fef7e1ca18cb00cf74c9cefbfb2e36fba26"} Jul 15 10:24:32.610339 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:32.610296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:32.991711 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:32.991652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:33.098904 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:33.098876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:33.255570 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:33.255504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:33.355158 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:33.355130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:33.475981 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:33.475954 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" event={"ID":"e68f98d8-e417-41cf-8c26-4dd54af6229a","Type":"ContainerStarted","Data":"79021bd405394f0723d6de4de8bb91da4be27d488d124b0249ea2e2a8d1d7d6d"} Jul 15 10:24:33.476097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:33.476076 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" Jul 15 10:24:34.657556 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.657510 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" podStartSLOduration=2.9421937849999997 podStartE2EDuration="5.657493349s" totalImagesPullingTime="2.715299564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:24:33.515900707 +0000 UTC m=+275.649094055" watchObservedRunningTime="2026-07-15 10:24:34.657493349 +0000 UTC m=+276.790686689" Jul 15 10:24:34.658600 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.658581 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2"] Jul 15 10:24:34.661808 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.661791 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" Jul 15 10:24:34.664204 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.664182 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 10:24:34.664296 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.664266 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 10:24:34.665093 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.665074 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-q2xsl\"" Jul 15 10:24:34.685955 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.685936 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2"] Jul 15 10:24:34.779060 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.779037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d55a01d-6152-4878-8db5-082e0970b2e4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6qxv2\" (UID: \"1d55a01d-6152-4878-8db5-082e0970b2e4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" Jul 15 10:24:34.779153 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.779066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bkd\" (UniqueName: \"kubernetes.io/projected/1d55a01d-6152-4878-8db5-082e0970b2e4-kube-api-access-f7bkd\") pod \"openshift-lws-operator-fd8ccff4c-6qxv2\" (UID: \"1d55a01d-6152-4878-8db5-082e0970b2e4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" Jul 15 10:24:34.879951 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.879928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d55a01d-6152-4878-8db5-082e0970b2e4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6qxv2\" (UID: \"1d55a01d-6152-4878-8db5-082e0970b2e4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" Jul 15 10:24:34.880068 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.879963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bkd\" (UniqueName: \"kubernetes.io/projected/1d55a01d-6152-4878-8db5-082e0970b2e4-kube-api-access-f7bkd\") pod \"openshift-lws-operator-fd8ccff4c-6qxv2\" (UID: \"1d55a01d-6152-4878-8db5-082e0970b2e4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" Jul 15 10:24:34.880270 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.880250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d55a01d-6152-4878-8db5-082e0970b2e4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6qxv2\" (UID: \"1d55a01d-6152-4878-8db5-082e0970b2e4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" Jul 15 10:24:34.891685 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.891656 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bkd\" (UniqueName: \"kubernetes.io/projected/1d55a01d-6152-4878-8db5-082e0970b2e4-kube-api-access-f7bkd\") pod \"openshift-lws-operator-fd8ccff4c-6qxv2\" (UID: \"1d55a01d-6152-4878-8db5-082e0970b2e4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" Jul 15 10:24:34.970231 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:34.970182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" Jul 15 10:24:35.108782 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:35.108731 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2"] Jul 15 10:24:35.112107 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:24:35.112085 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d55a01d_6152_4878_8db5_082e0970b2e4.slice/crio-4429876b4b928f91318bf2c5d3459feade3ba18263aa61a32107682f67621d36 WatchSource:0}: Error finding container 4429876b4b928f91318bf2c5d3459feade3ba18263aa61a32107682f67621d36: Status 404 returned error can't find the container with id 4429876b4b928f91318bf2c5d3459feade3ba18263aa61a32107682f67621d36 Jul 15 10:24:35.487529 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:35.487501 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" event={"ID":"1d55a01d-6152-4878-8db5-082e0970b2e4","Type":"ContainerStarted","Data":"4429876b4b928f91318bf2c5d3459feade3ba18263aa61a32107682f67621d36"} Jul 15 10:24:38.502323 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:38.502288 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" event={"ID":"1d55a01d-6152-4878-8db5-082e0970b2e4","Type":"ContainerStarted","Data":"4c5b1d158781413aaf2354dea01b2471a66d3f8615c103a9db10a10f6e91aa8b"} Jul 15 10:24:39.435606 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:39.435574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:39.482556 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:39.482526 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bgpl4" Jul 15 10:24:39.557530 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:39.557478 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6qxv2" podStartSLOduration=3.142510987 podStartE2EDuration="5.557459364s" totalImagesPullingTime="2.414948377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:24:38.583660334 +0000 UTC m=+280.716853672" watchObservedRunningTime="2026-07-15 10:24:39.557459364 +0000 UTC m=+281.690652706" Jul 15 10:24:39.580169 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:39.580133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:41.602401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:41.602361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:52.976938 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:52.976902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:53.087138 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:53.087093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:53.590246 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:53.590213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:53.934077 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:53.934040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:54.069617 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.069582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:54.151821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.151789 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg"] Jul 15 10:24:54.200154 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.200099 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg"] Jul 15 10:24:54.200281 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.200205 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.205031 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.205014 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 10:24:54.205449 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.205406 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 10:24:54.205449 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.205425 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xgzvh\"" Jul 15 10:24:54.205640 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.205495 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 10:24:54.205826 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.205813 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 10:24:54.218919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.218893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-tmp\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.219032 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.218929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.219032 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.219002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-manifests\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.219111 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.219045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.219111 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.219076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtjk\" (UniqueName: \"kubernetes.io/projected/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-kube-api-access-hmtjk\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.320265 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.320243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-tmp\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.320391 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.320284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.320391 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.320319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-manifests\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.320391 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.320357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.320546 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.320394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtjk\" (UniqueName: \"kubernetes.io/projected/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-kube-api-access-hmtjk\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.320731 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.320704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-manifests\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.322629 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.322606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-tmp\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.322863 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.322844 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.323036 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.323018 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.332602 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.332576 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtjk\" (UniqueName: \"kubernetes.io/projected/3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1-kube-api-access-hmtjk\") pod \"opendatahub-operator-controller-manager-656947bb4-44wmg\" (UID: \"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1\") " pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.338917 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.338883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:54.441862 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.441838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:54.510664 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.510607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:24:54.541073 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.541027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:54.630243 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.630214 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:54.649516 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:54.649492 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg"] Jul 15 10:24:54.653727 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:24:54.653702 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0f2f2b_1d6a_4723_9f23_3495ed57a9b1.slice/crio-4d191afc166490781bcb29a6532c338d42122c82d18bbc417889da1bba09413e WatchSource:0}: Error finding container 4d191afc166490781bcb29a6532c338d42122c82d18bbc417889da1bba09413e: Status 404 returned error can't find the container with id 4d191afc166490781bcb29a6532c338d42122c82d18bbc417889da1bba09413e Jul 15 10:24:55.574576 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:55.574532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" event={"ID":"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1","Type":"ContainerStarted","Data":"4d191afc166490781bcb29a6532c338d42122c82d18bbc417889da1bba09413e"} Jul 15 10:24:56.824827 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:56.824793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:56.915944 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:56.915900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:24:58.229813 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:58.229781 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:24:58.234947 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:58.234924 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:24:58.238059 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:58.238039 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 10:24:58.589533 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:58.589442 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" event={"ID":"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1","Type":"ContainerStarted","Data":"ac71218c99956f97645294bc43a25cfb309c87d08e8f87af22ff57de9843680a"} Jul 15 10:24:58.918462 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:24:58.918434 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0f2f2b_1d6a_4723_9f23_3495ed57a9b1.slice/crio-conmon-ac71218c99956f97645294bc43a25cfb309c87d08e8f87af22ff57de9843680a.scope\": RecentStats: unable to find data in memory cache]" Jul 15 10:24:59.595409 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:59.595376 2579 generic.go:356] "Generic (PLEG): container finished" podID="3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1" containerID="ac71218c99956f97645294bc43a25cfb309c87d08e8f87af22ff57de9843680a" exitCode=0 Jul 15 10:24:59.595814 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:59.595431 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" event={"ID":"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1","Type":"ContainerDied","Data":"ac71218c99956f97645294bc43a25cfb309c87d08e8f87af22ff57de9843680a"} Jul 15 10:24:59.595889 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:24:59.595873 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 10:25:00.601122 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:00.601086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" event={"ID":"3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1","Type":"ContainerStarted","Data":"6c04378d587fab944f0a3941ff2841aff4d1a3e19062ac366489c66e45949ba5"} Jul 15 10:25:00.601598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:00.601208 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:25:00.641197 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:00.641144 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" podStartSLOduration=2.083421863 podStartE2EDuration="6.641124929s" totalImagesPullingTime="3.492651221s" totalInitContainerRuntime="1.065051845s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:24:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:25:00.639688394 +0000 UTC m=+302.772881735" watchObservedRunningTime="2026-07-15 10:25:00.641124929 +0000 UTC m=+302.774318268" Jul 15 10:25:08.494460 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:08.494421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:09.889172 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:09.889136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:11.608372 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:11.608342 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-656947bb4-44wmg" Jul 15 10:25:14.856127 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:14.856100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:14.987830 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:14.987776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:15.506040 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.506007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:15.558116 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.558090 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv"] Jul 15 10:25:15.565892 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.565872 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.568453 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.568431 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 10:25:15.568685 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.568670 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 10:25:15.568767 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.568698 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 10:25:15.569455 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.569437 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 10:25:15.569535 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.569455 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lstmh\"" Jul 15 10:25:15.583120 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.583099 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv"] Jul 15 10:25:15.671277 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.671257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88a54609-e5a2-47ed-b449-1bb7588e6bc3-tmp\") pod \"kube-auth-proxy-544bf6b555-8zcfv\" (UID: \"88a54609-e5a2-47ed-b449-1bb7588e6bc3\") " pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.671392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.671307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a54609-e5a2-47ed-b449-1bb7588e6bc3-tls-certs\") pod \"kube-auth-proxy-544bf6b555-8zcfv\" (UID: \"88a54609-e5a2-47ed-b449-1bb7588e6bc3\") " pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.671392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.671352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qvx\" (UniqueName: \"kubernetes.io/projected/88a54609-e5a2-47ed-b449-1bb7588e6bc3-kube-api-access-88qvx\") pod \"kube-auth-proxy-544bf6b555-8zcfv\" (UID: \"88a54609-e5a2-47ed-b449-1bb7588e6bc3\") " pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.772385 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.772325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qvx\" (UniqueName: \"kubernetes.io/projected/88a54609-e5a2-47ed-b449-1bb7588e6bc3-kube-api-access-88qvx\") pod \"kube-auth-proxy-544bf6b555-8zcfv\" (UID: \"88a54609-e5a2-47ed-b449-1bb7588e6bc3\") " pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.772385 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.772358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88a54609-e5a2-47ed-b449-1bb7588e6bc3-tmp\") pod \"kube-auth-proxy-544bf6b555-8zcfv\" (UID: \"88a54609-e5a2-47ed-b449-1bb7588e6bc3\") " pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.772519 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.772487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a54609-e5a2-47ed-b449-1bb7588e6bc3-tls-certs\") pod \"kube-auth-proxy-544bf6b555-8zcfv\" (UID: \"88a54609-e5a2-47ed-b449-1bb7588e6bc3\") " pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.774758 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.774720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88a54609-e5a2-47ed-b449-1bb7588e6bc3-tmp\") pod \"kube-auth-proxy-544bf6b555-8zcfv\" (UID: \"88a54609-e5a2-47ed-b449-1bb7588e6bc3\") " pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.774916 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.774899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a54609-e5a2-47ed-b449-1bb7588e6bc3-tls-certs\") pod \"kube-auth-proxy-544bf6b555-8zcfv\" (UID: \"88a54609-e5a2-47ed-b449-1bb7588e6bc3\") " pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.785366 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.785342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qvx\" (UniqueName: \"kubernetes.io/projected/88a54609-e5a2-47ed-b449-1bb7588e6bc3-kube-api-access-88qvx\") pod \"kube-auth-proxy-544bf6b555-8zcfv\" (UID: \"88a54609-e5a2-47ed-b449-1bb7588e6bc3\") " pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.822687 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.822664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:15.875656 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.875636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" Jul 15 10:25:15.921210 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:15.921169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:16.015547 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:16.015523 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv"] Jul 15 10:25:16.016844 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:16.016816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:16.018332 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:25:16.018305 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a54609_e5a2_47ed_b449_1bb7588e6bc3.slice/crio-187c5a3694a1c264a5332aaa1d56c5ca5b5e88898eb79622566200c0f7d7b885 WatchSource:0}: Error finding container 187c5a3694a1c264a5332aaa1d56c5ca5b5e88898eb79622566200c0f7d7b885: Status 404 returned error can't find the container with id 187c5a3694a1c264a5332aaa1d56c5ca5b5e88898eb79622566200c0f7d7b885 Jul 15 10:25:16.136696 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:16.136667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:16.288944 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:16.288896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:16.667470 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:16.667435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" event={"ID":"88a54609-e5a2-47ed-b449-1bb7588e6bc3","Type":"ContainerStarted","Data":"187c5a3694a1c264a5332aaa1d56c5ca5b5e88898eb79622566200c0f7d7b885"} Jul 15 10:25:17.386076 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:17.386041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:17.507825 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:17.507783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:17.600366 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:17.600335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:18.932337 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:18.932296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:19.439458 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:19.439426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:19.504407 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:19.504377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:20.686964 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:20.686929 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" event={"ID":"88a54609-e5a2-47ed-b449-1bb7588e6bc3","Type":"ContainerStarted","Data":"61ba0f1bb2cdb0e503dcd6faaaf7ee886c7994fe6d5b3977bc445925ab0b3760"} Jul 15 10:25:20.713842 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:20.713799 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-544bf6b555-8zcfv" podStartSLOduration=1.614822075 podStartE2EDuration="5.713776509s" totalImagesPullingTime="4.098954434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:25:20.707033924 +0000 UTC m=+322.840227263" watchObservedRunningTime="2026-07-15 10:25:20.713776509 +0000 UTC m=+322.846969846" Jul 15 10:25:21.797538 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:21.797500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:21.896493 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:21.896455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:22.510363 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.510331 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc"] Jul 15 10:25:22.512494 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.512479 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.516903 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.516878 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 10:25:22.520269 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.520232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:22.523735 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.523718 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-49ptq\"" Jul 15 10:25:22.532109 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.532089 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc"] Jul 15 10:25:22.628618 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.628591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f1570a-64b2-4e3b-85d2-5470f77ba856-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.628730 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.628631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.628730 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.628698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.628879 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.628727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpn2\" (UniqueName: \"kubernetes.io/projected/95f1570a-64b2-4e3b-85d2-5470f77ba856-kube-api-access-4zpn2\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.628879 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.628787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.628879 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.628825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.629068 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.628884 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.629068 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.628926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.629068 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.628960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.729486 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.729461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.729581 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.729492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpn2\" (UniqueName: \"kubernetes.io/projected/95f1570a-64b2-4e3b-85d2-5470f77ba856-kube-api-access-4zpn2\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.729581 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.729511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.729581 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.729531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.729581 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.729557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.729581 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.729576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.729861 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.729606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.729861 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.729631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f1570a-64b2-4e3b-85d2-5470f77ba856-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.729861 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.729663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.730045 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.730022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.730259 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.730234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.730371 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.730349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.730482 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.730406 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.730811 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.730786 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f1570a-64b2-4e3b-85d2-5470f77ba856-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.731850 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.731831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.732252 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.732230 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.739874 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.739850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f1570a-64b2-4e3b-85d2-5470f77ba856-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.739975 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.739940 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpn2\" (UniqueName: \"kubernetes.io/projected/95f1570a-64b2-4e3b-85d2-5470f77ba856-kube-api-access-4zpn2\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67897rvc\" (UID: \"95f1570a-64b2-4e3b-85d2-5470f77ba856\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.822583 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.822525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:22.872813 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.872769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:22.955717 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.955640 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc"] Jul 15 10:25:22.957812 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:25:22.957789 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f1570a_64b2_4e3b_85d2_5470f77ba856.slice/crio-3ee55bd28d93abe0f73abe9f5450795e21670c09f2fce0294f3f4011ffe7b05f WatchSource:0}: Error finding container 3ee55bd28d93abe0f73abe9f5450795e21670c09f2fce0294f3f4011ffe7b05f: Status 404 returned error can't find the container with id 3ee55bd28d93abe0f73abe9f5450795e21670c09f2fce0294f3f4011ffe7b05f Jul 15 10:25:22.959129 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:22.959086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:23.107941 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:23.107899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:23.700148 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:23.700111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" event={"ID":"95f1570a-64b2-4e3b-85d2-5470f77ba856","Type":"ContainerStarted","Data":"3ee55bd28d93abe0f73abe9f5450795e21670c09f2fce0294f3f4011ffe7b05f"} Jul 15 10:25:24.147140 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.147104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:24.207702 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.207673 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd"] Jul 15 10:25:24.210650 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.210634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.213207 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.213184 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 10:25:24.214432 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.214416 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 10:25:24.214432 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.214425 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 10:25:24.214575 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.214425 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mm624\"" Jul 15 10:25:24.233073 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.233052 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd"] Jul 15 10:25:24.343225 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.343194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-cert\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.343225 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.343226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-manager-config\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.343426 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.343242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-metrics-cert\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.343426 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.343262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cl8t\" (UniqueName: \"kubernetes.io/projected/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-kube-api-access-9cl8t\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.443994 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.443912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-cert\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.443994 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.443951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-manager-config\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.443994 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.443972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-metrics-cert\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.444236 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.443995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cl8t\" (UniqueName: \"kubernetes.io/projected/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-kube-api-access-9cl8t\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.444624 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.444605 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-manager-config\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.446677 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.446649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-metrics-cert\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.446814 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.446719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-cert\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.455334 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.455309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cl8t\" (UniqueName: \"kubernetes.io/projected/61806022-8dd9-48d9-bd55-8aaa6b5a9f4a-kube-api-access-9cl8t\") pod \"lws-controller-manager-587d8d67f-6t8rd\" (UID: \"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a\") " pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.522450 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.522426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:24.687659 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.687548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd"] Jul 15 10:25:24.691432 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:25:24.691403 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61806022_8dd9_48d9_bd55_8aaa6b5a9f4a.slice/crio-38e128337b62c8671cb3d4e3270ef3401439f5c929b1adf4b7872335bcf32fa8 WatchSource:0}: Error finding container 38e128337b62c8671cb3d4e3270ef3401439f5c929b1adf4b7872335bcf32fa8: Status 404 returned error can't find the container with id 38e128337b62c8671cb3d4e3270ef3401439f5c929b1adf4b7872335bcf32fa8 Jul 15 10:25:24.706925 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:24.706888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" event={"ID":"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a","Type":"ContainerStarted","Data":"38e128337b62c8671cb3d4e3270ef3401439f5c929b1adf4b7872335bcf32fa8"} Jul 15 10:25:26.626499 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:26.626463 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:25:26.626754 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:26.626531 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:25:26.626754 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:26.626559 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:25:26.718837 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:26.718797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" event={"ID":"95f1570a-64b2-4e3b-85d2-5470f77ba856","Type":"ContainerStarted","Data":"37cec608eb72c429a84c18e9518566e65fb9a53597fc758897fa93a99c06229b"} Jul 15 10:25:26.770416 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:26.770357 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" podStartSLOduration=1.103978694 podStartE2EDuration="4.770336904s" totalImagesPullingTime="3.66635821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:25:26.769734132 +0000 UTC m=+328.902927471" watchObservedRunningTime="2026-07-15 10:25:26.770336904 +0000 UTC m=+328.903530242" Jul 15 10:25:26.823174 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:26.823136 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:26.954332 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:26.954253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:27.051322 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:27.051273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:27.168578 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:27.168549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:27.380373 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:27.380342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:27.470856 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:27.470821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:27.571143 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:27.571116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:25:27.827874 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:27.827799 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:28.728503 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:28.728466 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" event={"ID":"61806022-8dd9-48d9-bd55-8aaa6b5a9f4a","Type":"ContainerStarted","Data":"9201a4ee536e35e537f024e8749b73da68797b2d077fe9d74e1b35d553a15791"} Jul 15 10:25:28.728730 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:28.728527 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:25:28.728809 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:28.728776 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:28.729771 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:28.729735 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67897rvc" Jul 15 10:25:28.748171 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:28.748133 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" podStartSLOduration=1.530862216 podStartE2EDuration="4.748121105s" totalImagesPullingTime="3.217258889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:25:28.746658833 +0000 UTC m=+330.879852170" watchObservedRunningTime="2026-07-15 10:25:28.748121105 +0000 UTC m=+330.881314442" Jul 15 10:25:39.735184 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:25:39.735153 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-587d8d67f-6t8rd" Jul 15 10:26:02.421540 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:02.421444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:02.514848 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:02.514814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:16.341066 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:16.341024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:16.436522 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:16.436483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:16.616540 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:16.616505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:35.851154 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:35.851114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:35.923497 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:35.923461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:39.633631 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:39.633588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:45.676472 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:45.676437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:47.144327 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:47.144289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:47.271495 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:47.271457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:47.376407 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:47.376358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:49.699304 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:49.699269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:52.275512 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:52.275482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:53.908201 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:53.908168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:55.314601 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:55.314567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:56.889603 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:56.889571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:57.126969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:57.126931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:57.481903 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:57.481852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:59.708896 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:59.708865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:26:59.807660 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:26:59.807625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:01.894295 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:01.894260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:01.979125 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:01.979066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:03.885357 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:03.885326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:03.986201 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:03.986172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:06.980038 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:06.979999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:07.227244 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:07.227215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:09.954217 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:09.954170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:10.055128 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:10.055097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:17.514475 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:17.514438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:17.615501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:17.615475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:17.722818 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:17.722785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:20.096705 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:20.096629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:20.198691 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:20.198663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:22.650945 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:22.650914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:23.872919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:23.872889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:26.788402 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:26.788369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:26.858280 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:26.858249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:35.357348 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:35.357312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:35.457326 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:35.457296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:57.771894 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:57.771864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:57.893110 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:57.893082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:58.831027 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:58.830990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:27:58.931849 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:27:58.931822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:12.956810 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:12.956768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:13.074717 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.074680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:13.135641 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.135612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:13.832209 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.832179 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-k5c7d"] Jul 15 10:28:13.835287 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.835270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-k5c7d" Jul 15 10:28:13.837673 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.837650 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 10:28:13.838466 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.838451 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2tsn5\"" Jul 15 10:28:13.838532 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.838482 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 10:28:13.845029 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.845004 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-k5c7d"] Jul 15 10:28:13.867573 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.867552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9qgz\" (UniqueName: \"kubernetes.io/projected/7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0-kube-api-access-q9qgz\") pod \"authorino-58cd955675-k5c7d\" (UID: \"7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0\") " pod="kuadrant-system/authorino-58cd955675-k5c7d" Jul 15 10:28:13.868346 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.868328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:13.930760 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.930721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:13.967977 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.967958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qgz\" (UniqueName: \"kubernetes.io/projected/7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0-kube-api-access-q9qgz\") pod \"authorino-58cd955675-k5c7d\" (UID: \"7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0\") " pod="kuadrant-system/authorino-58cd955675-k5c7d" Jul 15 10:28:13.977782 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.977762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qgz\" (UniqueName: \"kubernetes.io/projected/7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0-kube-api-access-q9qgz\") pod \"authorino-58cd955675-k5c7d\" (UID: \"7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0\") " pod="kuadrant-system/authorino-58cd955675-k5c7d" Jul 15 10:28:13.992363 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.992343 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-ngqvd"] Jul 15 10:28:13.995470 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:13.995454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ngqvd" Jul 15 10:28:14.005454 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:14.005433 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ngqvd"] Jul 15 10:28:14.068665 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:14.068638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9bt6\" (UniqueName: \"kubernetes.io/projected/f4728114-3c14-40a9-a150-bbbece1963de-kube-api-access-b9bt6\") pod \"authorino-75d556bd77-ngqvd\" (UID: \"f4728114-3c14-40a9-a150-bbbece1963de\") " pod="kuadrant-system/authorino-75d556bd77-ngqvd" Jul 15 10:28:14.145220 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:14.145199 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-k5c7d" Jul 15 10:28:14.169244 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:14.169222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bt6\" (UniqueName: \"kubernetes.io/projected/f4728114-3c14-40a9-a150-bbbece1963de-kube-api-access-b9bt6\") pod \"authorino-75d556bd77-ngqvd\" (UID: \"f4728114-3c14-40a9-a150-bbbece1963de\") " pod="kuadrant-system/authorino-75d556bd77-ngqvd" Jul 15 10:28:14.181665 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:14.181638 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9bt6\" (UniqueName: \"kubernetes.io/projected/f4728114-3c14-40a9-a150-bbbece1963de-kube-api-access-b9bt6\") pod \"authorino-75d556bd77-ngqvd\" (UID: \"f4728114-3c14-40a9-a150-bbbece1963de\") " pod="kuadrant-system/authorino-75d556bd77-ngqvd" Jul 15 10:28:14.275948 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:14.275922 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-k5c7d"] Jul 15 10:28:14.278975 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:28:14.278951 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c2c76d6_a8e6_4039_8b92_c958f1e6f9c0.slice/crio-2c68360c8c6d93496b8553321455dd3373b6fb9c15054ca0619952b2d3c2fe73 WatchSource:0}: Error finding container 2c68360c8c6d93496b8553321455dd3373b6fb9c15054ca0619952b2d3c2fe73: Status 404 returned error can't find the container with id 2c68360c8c6d93496b8553321455dd3373b6fb9c15054ca0619952b2d3c2fe73 Jul 15 10:28:14.303899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:14.303874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ngqvd" Jul 15 10:28:14.433468 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:14.433446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ngqvd"] Jul 15 10:28:14.435728 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:28:14.435702 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4728114_3c14_40a9_a150_bbbece1963de.slice/crio-91b98591dea818425718aab52cc9e3b32e005dac6d98a8e6f549341d7abfbb32 WatchSource:0}: Error finding container 91b98591dea818425718aab52cc9e3b32e005dac6d98a8e6f549341d7abfbb32: Status 404 returned error can't find the container with id 91b98591dea818425718aab52cc9e3b32e005dac6d98a8e6f549341d7abfbb32 Jul 15 10:28:14.447017 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:14.446983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-k5c7d" event={"ID":"7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0","Type":"ContainerStarted","Data":"2c68360c8c6d93496b8553321455dd3373b6fb9c15054ca0619952b2d3c2fe73"} Jul 15 10:28:14.448021 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:14.448003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ngqvd" event={"ID":"f4728114-3c14-40a9-a150-bbbece1963de","Type":"ContainerStarted","Data":"91b98591dea818425718aab52cc9e3b32e005dac6d98a8e6f549341d7abfbb32"} Jul 15 10:28:18.471264 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:18.471224 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-k5c7d" event={"ID":"7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0","Type":"ContainerStarted","Data":"d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005"} Jul 15 10:28:18.472600 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:18.472569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ngqvd" event={"ID":"f4728114-3c14-40a9-a150-bbbece1963de","Type":"ContainerStarted","Data":"cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640"} Jul 15 10:28:18.488015 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:18.487955 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-k5c7d" podStartSLOduration=1.889372454 podStartE2EDuration="5.487936848s" totalImagesPullingTime="3.598564394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:28:18.487554352 +0000 UTC m=+500.620747690" watchObservedRunningTime="2026-07-15 10:28:18.487936848 +0000 UTC m=+500.621130187" Jul 15 10:28:18.503483 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:18.503439 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-ngqvd" podStartSLOduration=2.072335171 podStartE2EDuration="5.503426264s" totalImagesPullingTime="3.431091093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:28:18.502032983 +0000 UTC m=+500.635226321" watchObservedRunningTime="2026-07-15 10:28:18.503426264 +0000 UTC m=+500.636619601" Jul 15 10:28:18.531877 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:18.531852 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-k5c7d"] Jul 15 10:28:20.481627 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:20.481588 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-k5c7d" podUID="7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0" containerName="authorino" containerID="cri-o://d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005" gracePeriod=30 Jul 15 10:28:20.721139 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:20.721107 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-k5c7d" Jul 15 10:28:20.844681 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:20.844656 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9qgz\" (UniqueName: \"kubernetes.io/projected/7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0-kube-api-access-q9qgz\") pod \"7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0\" (UID: \"7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0\") " Jul 15 10:28:20.846889 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:20.846863 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0-kube-api-access-q9qgz" (OuterVolumeSpecName: "kube-api-access-q9qgz") pod "7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0" (UID: "7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0"). InnerVolumeSpecName "kube-api-access-q9qgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:28:20.946030 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:20.946004 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9qgz\" (UniqueName: \"kubernetes.io/projected/7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0-kube-api-access-q9qgz\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:28:21.487598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:21.487570 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0" containerID="d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005" exitCode=0 Jul 15 10:28:21.488004 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:21.487611 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-k5c7d" Jul 15 10:28:21.488004 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:21.487615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-k5c7d" event={"ID":"7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0","Type":"ContainerDied","Data":"d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005"} Jul 15 10:28:21.488004 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:21.487639 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-k5c7d" event={"ID":"7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0","Type":"ContainerDied","Data":"2c68360c8c6d93496b8553321455dd3373b6fb9c15054ca0619952b2d3c2fe73"} Jul 15 10:28:21.488004 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:21.487653 2579 scope.go:117] "RemoveContainer" containerID="d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005" Jul 15 10:28:21.496346 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:21.496327 2579 scope.go:117] "RemoveContainer" containerID="d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005" Jul 15 10:28:21.496576 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:28:21.496559 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005\": container with ID starting with d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005 not found: ID does not exist" containerID="d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005" Jul 15 10:28:21.496635 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:21.496582 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005"} err="failed to get container status \"d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005\": rpc error: code = NotFound desc = could not find container \"d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005\": container with ID starting with d756bb5d8f1982009248cf92cd65c4415cc997074e6012ebaaa06295c353e005 not found: ID does not exist" Jul 15 10:28:21.510570 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:21.510545 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-k5c7d"] Jul 15 10:28:21.515406 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:21.515382 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-k5c7d"] Jul 15 10:28:22.350754 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:22.350705 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0" path="/var/lib/kubelet/pods/7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0/volumes" Jul 15 10:28:30.982110 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:30.982071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:31.082879 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:31.082849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:31.539039 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:31.539008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:43.907302 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:43.907268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:44.006633 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.006605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:44.156485 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.156460 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xcxtn"] Jul 15 10:28:44.156782 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.156770 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0" containerName="authorino" Jul 15 10:28:44.156831 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.156784 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0" containerName="authorino" Jul 15 10:28:44.156865 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.156850 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c2c76d6-a8e6-4039-8b92-c958f1e6f9c0" containerName="authorino" Jul 15 10:28:44.160832 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.160785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xcxtn" Jul 15 10:28:44.170428 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.170403 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xcxtn"] Jul 15 10:28:44.330034 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.330009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rkgc\" (UniqueName: \"kubernetes.io/projected/8229f4e0-c2e1-460c-a033-ccf781db133b-kube-api-access-5rkgc\") pod \"authorino-5d4b67c446-xcxtn\" (UID: \"8229f4e0-c2e1-460c-a033-ccf781db133b\") " pod="kuadrant-system/authorino-5d4b67c446-xcxtn" Jul 15 10:28:44.382911 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.382891 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xcxtn"] Jul 15 10:28:44.383061 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:28:44.383044 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5rkgc], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-xcxtn" podUID="8229f4e0-c2e1-460c-a033-ccf781db133b" Jul 15 10:28:44.424158 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.424105 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77456d549c-d8jr4"] Jul 15 10:28:44.426306 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.426289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77456d549c-d8jr4" Jul 15 10:28:44.428793 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.428771 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 10:28:44.430843 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.430824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rkgc\" (UniqueName: \"kubernetes.io/projected/8229f4e0-c2e1-460c-a033-ccf781db133b-kube-api-access-5rkgc\") pod \"authorino-5d4b67c446-xcxtn\" (UID: \"8229f4e0-c2e1-460c-a033-ccf781db133b\") " pod="kuadrant-system/authorino-5d4b67c446-xcxtn" Jul 15 10:28:44.443359 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.443338 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77456d549c-d8jr4"] Jul 15 10:28:44.450118 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.450100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rkgc\" (UniqueName: \"kubernetes.io/projected/8229f4e0-c2e1-460c-a033-ccf781db133b-kube-api-access-5rkgc\") pod \"authorino-5d4b67c446-xcxtn\" (UID: \"8229f4e0-c2e1-460c-a033-ccf781db133b\") " pod="kuadrant-system/authorino-5d4b67c446-xcxtn" Jul 15 10:28:44.532056 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.532030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxt9\" (UniqueName: \"kubernetes.io/projected/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-kube-api-access-twxt9\") pod \"authorino-77456d549c-d8jr4\" (UID: \"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b\") " pod="kuadrant-system/authorino-77456d549c-d8jr4" Jul 15 10:28:44.532146 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.532096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-tls-cert\") pod \"authorino-77456d549c-d8jr4\" (UID: \"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b\") " pod="kuadrant-system/authorino-77456d549c-d8jr4" Jul 15 10:28:44.587109 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.587085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xcxtn" Jul 15 10:28:44.593141 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.593123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xcxtn" Jul 15 10:28:44.632524 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.632500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-tls-cert\") pod \"authorino-77456d549c-d8jr4\" (UID: \"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b\") " pod="kuadrant-system/authorino-77456d549c-d8jr4" Jul 15 10:28:44.632607 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.632552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxt9\" (UniqueName: \"kubernetes.io/projected/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-kube-api-access-twxt9\") pod \"authorino-77456d549c-d8jr4\" (UID: \"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b\") " pod="kuadrant-system/authorino-77456d549c-d8jr4" Jul 15 10:28:44.634937 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.634911 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-tls-cert\") pod \"authorino-77456d549c-d8jr4\" (UID: \"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b\") " pod="kuadrant-system/authorino-77456d549c-d8jr4" Jul 15 10:28:44.642814 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.642797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxt9\" (UniqueName: \"kubernetes.io/projected/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-kube-api-access-twxt9\") pod \"authorino-77456d549c-d8jr4\" (UID: \"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b\") " pod="kuadrant-system/authorino-77456d549c-d8jr4" Jul 15 10:28:44.733551 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.733503 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rkgc\" (UniqueName: \"kubernetes.io/projected/8229f4e0-c2e1-460c-a033-ccf781db133b-kube-api-access-5rkgc\") pod \"8229f4e0-c2e1-460c-a033-ccf781db133b\" (UID: \"8229f4e0-c2e1-460c-a033-ccf781db133b\") " Jul 15 10:28:44.734864 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.734841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77456d549c-d8jr4" Jul 15 10:28:44.735455 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.735430 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8229f4e0-c2e1-460c-a033-ccf781db133b-kube-api-access-5rkgc" (OuterVolumeSpecName: "kube-api-access-5rkgc") pod "8229f4e0-c2e1-460c-a033-ccf781db133b" (UID: "8229f4e0-c2e1-460c-a033-ccf781db133b"). InnerVolumeSpecName "kube-api-access-5rkgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:28:44.834388 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.834352 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rkgc\" (UniqueName: \"kubernetes.io/projected/8229f4e0-c2e1-460c-a033-ccf781db133b-kube-api-access-5rkgc\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:28:44.860561 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:44.860540 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77456d549c-d8jr4"] Jul 15 10:28:44.863106 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:28:44.863084 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfa8f7b0_0c86_45f0_808f_6d1807a7c75b.slice/crio-57bb1b947c1117fa87d55b074a0f9b94ef84122ebdd98be0c2199447e8889e87 WatchSource:0}: Error finding container 57bb1b947c1117fa87d55b074a0f9b94ef84122ebdd98be0c2199447e8889e87: Status 404 returned error can't find the container with id 57bb1b947c1117fa87d55b074a0f9b94ef84122ebdd98be0c2199447e8889e87 Jul 15 10:28:45.593495 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:45.593400 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77456d549c-d8jr4" event={"ID":"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b","Type":"ContainerStarted","Data":"354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b"} Jul 15 10:28:45.593495 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:45.593452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xcxtn" Jul 15 10:28:45.593944 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:45.593457 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77456d549c-d8jr4" event={"ID":"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b","Type":"ContainerStarted","Data":"57bb1b947c1117fa87d55b074a0f9b94ef84122ebdd98be0c2199447e8889e87"} Jul 15 10:28:45.612161 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:45.612120 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77456d549c-d8jr4" podStartSLOduration=1.259491988 podStartE2EDuration="1.612105167s" totalImagesPullingTime="352.613179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:28:45.610517306 +0000 UTC m=+527.743710644" watchObservedRunningTime="2026-07-15 10:28:45.612105167 +0000 UTC m=+527.745298526" Jul 15 10:28:45.635476 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:45.635452 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xcxtn"] Jul 15 10:28:45.642323 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:45.642303 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xcxtn"] Jul 15 10:28:45.646773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:45.646734 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ngqvd"] Jul 15 10:28:45.646924 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:45.646904 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-ngqvd" podUID="f4728114-3c14-40a9-a150-bbbece1963de" containerName="authorino" containerID="cri-o://cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640" gracePeriod=30 Jul 15 10:28:45.888064 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:45.888043 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ngqvd" Jul 15 10:28:46.045524 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.045492 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9bt6\" (UniqueName: \"kubernetes.io/projected/f4728114-3c14-40a9-a150-bbbece1963de-kube-api-access-b9bt6\") pod \"f4728114-3c14-40a9-a150-bbbece1963de\" (UID: \"f4728114-3c14-40a9-a150-bbbece1963de\") " Jul 15 10:28:46.047617 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.047592 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4728114-3c14-40a9-a150-bbbece1963de-kube-api-access-b9bt6" (OuterVolumeSpecName: "kube-api-access-b9bt6") pod "f4728114-3c14-40a9-a150-bbbece1963de" (UID: "f4728114-3c14-40a9-a150-bbbece1963de"). InnerVolumeSpecName "kube-api-access-b9bt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:28:46.147155 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.147098 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9bt6\" (UniqueName: \"kubernetes.io/projected/f4728114-3c14-40a9-a150-bbbece1963de-kube-api-access-b9bt6\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:28:46.355883 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.353901 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8229f4e0-c2e1-460c-a033-ccf781db133b" path="/var/lib/kubelet/pods/8229f4e0-c2e1-460c-a033-ccf781db133b/volumes" Jul 15 10:28:46.599181 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.599144 2579 generic.go:356] "Generic (PLEG): container finished" podID="f4728114-3c14-40a9-a150-bbbece1963de" containerID="cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640" exitCode=0 Jul 15 10:28:46.599604 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.599208 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ngqvd" Jul 15 10:28:46.599604 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.599237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ngqvd" event={"ID":"f4728114-3c14-40a9-a150-bbbece1963de","Type":"ContainerDied","Data":"cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640"} Jul 15 10:28:46.599604 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.599278 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ngqvd" event={"ID":"f4728114-3c14-40a9-a150-bbbece1963de","Type":"ContainerDied","Data":"91b98591dea818425718aab52cc9e3b32e005dac6d98a8e6f549341d7abfbb32"} Jul 15 10:28:46.599604 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.599297 2579 scope.go:117] "RemoveContainer" containerID="cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640" Jul 15 10:28:46.607560 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.607543 2579 scope.go:117] "RemoveContainer" containerID="cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640" Jul 15 10:28:46.607820 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:28:46.607801 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640\": container with ID starting with cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640 not found: ID does not exist" containerID="cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640" Jul 15 10:28:46.607878 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.607829 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640"} err="failed to get container status \"cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640\": rpc error: code = NotFound desc = could not find container \"cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640\": container with ID starting with cb56725cad4953a2a43178def2afe4a6c866abaa40b773ebc2f4e3a1574dd640 not found: ID does not exist" Jul 15 10:28:46.616836 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.616812 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ngqvd"] Jul 15 10:28:46.619345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:46.619325 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ngqvd"] Jul 15 10:28:48.350505 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:48.350474 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4728114-3c14-40a9-a150-bbbece1963de" path="/var/lib/kubelet/pods/f4728114-3c14-40a9-a150-bbbece1963de/volumes" Jul 15 10:28:50.420041 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:50.420007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:50.522501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:50.522469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:50.625187 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:50.625134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:50.779271 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:50.779164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:54.319898 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:54.319859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:54.420093 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:54.420063 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:28:56.188845 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:28:56.188812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:02.513695 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:02.513634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:03.563713 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:03.563672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:03.628511 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:03.628476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:25.841522 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:25.841486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:26.086732 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:26.086696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:26.330063 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:26.330026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:26.415649 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:26.415622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:26.916643 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:26.916606 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:27.017335 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:27.017308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:37.884274 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:37.884237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:37.974550 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:37.974508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:46.216837 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:46.216802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:29:58.267604 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:58.267569 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:29:58.269686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:29:58.269658 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:30:00.145984 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.145950 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx"] Jul 15 10:30:00.148226 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.146265 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4728114-3c14-40a9-a150-bbbece1963de" containerName="authorino" Jul 15 10:30:00.148226 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.146276 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4728114-3c14-40a9-a150-bbbece1963de" containerName="authorino" Jul 15 10:30:00.148226 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.146338 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4728114-3c14-40a9-a150-bbbece1963de" containerName="authorino" Jul 15 10:30:00.149049 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.149033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx" Jul 15 10:30:00.151623 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.151604 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zj826\"" Jul 15 10:30:00.158009 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.157987 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx"] Jul 15 10:30:00.159666 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.159646 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx" Jul 15 10:30:00.292917 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.292883 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx"] Jul 15 10:30:00.296034 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:30:00.296006 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d06b3e_cbba_4b70_b396_50d5e3208576.slice/crio-8a4b95451ab40525cfc451abaf160d771449c9ea9ba188f6a41308b64a5ad54a WatchSource:0}: Error finding container 8a4b95451ab40525cfc451abaf160d771449c9ea9ba188f6a41308b64a5ad54a: Status 404 returned error can't find the container with id 8a4b95451ab40525cfc451abaf160d771449c9ea9ba188f6a41308b64a5ad54a Jul 15 10:30:00.298361 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.298345 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 10:30:00.932853 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:00.932813 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx" event={"ID":"39d06b3e-cbba-4b70-b396-50d5e3208576","Type":"ContainerStarted","Data":"8a4b95451ab40525cfc451abaf160d771449c9ea9ba188f6a41308b64a5ad54a"} Jul 15 10:30:03.377054 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.377023 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg"] Jul 15 10:30:03.381547 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.381526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.385033 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.385011 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 15 10:30:03.385033 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.385026 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lmmjs\"" Jul 15 10:30:03.385192 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.385011 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 10:30:03.385192 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.385013 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 10:30:03.390106 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.390051 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg"] Jul 15 10:30:03.415358 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.415336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:30:03.524298 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.524275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.524397 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.524302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.524397 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.524332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.524397 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.524363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.524501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.524398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cqv\" (UniqueName: \"kubernetes.io/projected/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-kube-api-access-p4cqv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.524501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.524442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.624753 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.624714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cqv\" (UniqueName: \"kubernetes.io/projected/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-kube-api-access-p4cqv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.624860 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.624766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.624860 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.624794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.624860 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.624814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.624860 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.624856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.625085 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.624896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.625192 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.625172 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.625253 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.625200 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.625253 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.625218 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.627033 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.627010 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.627266 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.627250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.636171 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.636147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cqv\" (UniqueName: \"kubernetes.io/projected/c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d-kube-api-access-p4cqv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5trg\" (UID: \"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.698015 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.697990 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:03.832589 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.832568 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg"] Jul 15 10:30:03.834717 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:30:03.834692 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c8e7f3_b8b5_49f6_bb0f_7660ac5fc33d.slice/crio-2a7e1824ec39b1829e70088afb6588ffac2526a715d9ca4fa22ffafd31d0d2be WatchSource:0}: Error finding container 2a7e1824ec39b1829e70088afb6588ffac2526a715d9ca4fa22ffafd31d0d2be: Status 404 returned error can't find the container with id 2a7e1824ec39b1829e70088afb6588ffac2526a715d9ca4fa22ffafd31d0d2be Jul 15 10:30:03.951538 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.951503 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx" event={"ID":"39d06b3e-cbba-4b70-b396-50d5e3208576","Type":"ContainerStarted","Data":"3aa239b96268f1a06bcff629935bf3559b9adb83ed935dce1d5845424fe1274f"} Jul 15 10:30:03.952708 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.952685 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" event={"ID":"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d","Type":"ContainerStarted","Data":"2a7e1824ec39b1829e70088afb6588ffac2526a715d9ca4fa22ffafd31d0d2be"} Jul 15 10:30:03.967913 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:03.967876 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx" podStartSLOduration=1.044177592 podStartE2EDuration="3.967858458s" totalImagesPullingTime="2.923680866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:30:03.967172351 +0000 UTC m=+606.100365691" watchObservedRunningTime="2026-07-15 10:30:03.967858458 +0000 UTC m=+606.101051796" Jul 15 10:30:04.961606 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:04.961562 2579 generic.go:356] "Generic (PLEG): container finished" podID="39d06b3e-cbba-4b70-b396-50d5e3208576" containerID="3aa239b96268f1a06bcff629935bf3559b9adb83ed935dce1d5845424fe1274f" exitCode=0 Jul 15 10:30:04.962074 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:04.961631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx" event={"ID":"39d06b3e-cbba-4b70-b396-50d5e3208576","Type":"ContainerDied","Data":"3aa239b96268f1a06bcff629935bf3559b9adb83ed935dce1d5845424fe1274f"} Jul 15 10:30:05.968456 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:05.968420 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" event={"ID":"c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d","Type":"ContainerStarted","Data":"9e78c959e342ecfb3b10e19350f45b63fc3a034c16c5b6f988f8fc324765be27"} Jul 15 10:30:06.094174 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:06.094151 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx" Jul 15 10:30:06.114878 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:06.114819 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" podStartSLOduration=1.847262825 podStartE2EDuration="3.114797948s" totalImagesPullingTime="1.267535123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:30:05.987864365 +0000 UTC m=+608.121057703" watchObservedRunningTime="2026-07-15 10:30:06.114797948 +0000 UTC m=+608.247991288" Jul 15 10:30:06.973497 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:06.973471 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx" Jul 15 10:30:06.973919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:06.973519 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735190-twfmx" event={"ID":"39d06b3e-cbba-4b70-b396-50d5e3208576","Type":"ContainerDied","Data":"8a4b95451ab40525cfc451abaf160d771449c9ea9ba188f6a41308b64a5ad54a"} Jul 15 10:30:06.973919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:06.973563 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a4b95451ab40525cfc451abaf160d771449c9ea9ba188f6a41308b64a5ad54a" Jul 15 10:30:13.699202 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:13.699165 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:13.699645 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:13.699269 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:13.711292 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:13.711264 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:14.019696 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:14.019623 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5trg" Jul 15 10:30:14.722704 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:14.722670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:30:26.126804 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:26.126700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:30:33.893527 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.893492 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86"] Jul 15 10:30:33.894111 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.893985 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d06b3e-cbba-4b70-b396-50d5e3208576" containerName="cleanup" Jul 15 10:30:33.894111 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.894003 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d06b3e-cbba-4b70-b396-50d5e3208576" containerName="cleanup" Jul 15 10:30:33.894111 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.894104 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d06b3e-cbba-4b70-b396-50d5e3208576" containerName="cleanup" Jul 15 10:30:33.899169 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.899147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:33.901862 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.901830 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 10:30:33.907626 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.907606 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86"] Jul 15 10:30:33.957305 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.957280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:33.957406 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.957320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427624df-65e0-404c-9fd5-72d2ca92c989-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:33.957406 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.957371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:33.957540 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.957423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h56r\" (UniqueName: \"kubernetes.io/projected/427624df-65e0-404c-9fd5-72d2ca92c989-kube-api-access-2h56r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:33.957540 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.957467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:33.957540 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.957495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:33.973298 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:33.973276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:30:34.058340 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.058316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.058456 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.058346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427624df-65e0-404c-9fd5-72d2ca92c989-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.058456 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.058375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.058456 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.058406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h56r\" (UniqueName: \"kubernetes.io/projected/427624df-65e0-404c-9fd5-72d2ca92c989-kube-api-access-2h56r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.058456 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.058440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.058667 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.058466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.058760 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.058708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.058826 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.058774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.058882 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.058846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.060852 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.060831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427624df-65e0-404c-9fd5-72d2ca92c989-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.061259 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.061229 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427624df-65e0-404c-9fd5-72d2ca92c989-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.069358 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.069317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h56r\" (UniqueName: \"kubernetes.io/projected/427624df-65e0-404c-9fd5-72d2ca92c989-kube-api-access-2h56r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gjm86\" (UID: \"427624df-65e0-404c-9fd5-72d2ca92c989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.210407 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.210353 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:34.344624 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:34.344545 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86"] Jul 15 10:30:34.347317 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:30:34.347290 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427624df_65e0_404c_9fd5_72d2ca92c989.slice/crio-4283bdeec2b6872f6a5bc13ecf4a3b7165c2a939092d82eed896d2845d611951 WatchSource:0}: Error finding container 4283bdeec2b6872f6a5bc13ecf4a3b7165c2a939092d82eed896d2845d611951: Status 404 returned error can't find the container with id 4283bdeec2b6872f6a5bc13ecf4a3b7165c2a939092d82eed896d2845d611951 Jul 15 10:30:35.109897 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:35.109863 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" event={"ID":"427624df-65e0-404c-9fd5-72d2ca92c989","Type":"ContainerStarted","Data":"590f04ef308e052e4e9440a1201e3ddf1358e69d1ddfe996cbb6a23c8a05136d"} Jul 15 10:30:35.110261 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:35.109915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" event={"ID":"427624df-65e0-404c-9fd5-72d2ca92c989","Type":"ContainerStarted","Data":"4283bdeec2b6872f6a5bc13ecf4a3b7165c2a939092d82eed896d2845d611951"} Jul 15 10:30:35.134201 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:35.134157 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" podStartSLOduration=1.951915464 podStartE2EDuration="2.134141797s" totalImagesPullingTime="182.226333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:30:35.132711568 +0000 UTC m=+637.265904911" watchObservedRunningTime="2026-07-15 10:30:35.134141797 +0000 UTC m=+637.267335134" Jul 15 10:30:41.287582 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.287550 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p"] Jul 15 10:30:41.290949 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.290932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.293851 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.293825 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 10:30:41.302465 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.302442 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p"] Jul 15 10:30:41.320411 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.320387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:30:41.413943 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.413921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7nh\" (UniqueName: \"kubernetes.io/projected/df7cc2c0-554d-4cb6-a830-27d7d0282252-kube-api-access-5w7nh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.414055 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.413949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.414055 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.413972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.414055 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.413988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.414055 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.414046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.414228 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.414161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7cc2c0-554d-4cb6-a830-27d7d0282252-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.514876 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.514847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7cc2c0-554d-4cb6-a830-27d7d0282252-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.514989 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.514939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7nh\" (UniqueName: \"kubernetes.io/projected/df7cc2c0-554d-4cb6-a830-27d7d0282252-kube-api-access-5w7nh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.514989 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.514969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.515091 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.514997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.515091 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.515020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.515091 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.515047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.515408 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.515390 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.515484 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.515431 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.515540 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.515494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.517448 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.517429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7cc2c0-554d-4cb6-a830-27d7d0282252-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.517609 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.517592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7cc2c0-554d-4cb6-a830-27d7d0282252-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.524803 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.524726 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7nh\" (UniqueName: \"kubernetes.io/projected/df7cc2c0-554d-4cb6-a830-27d7d0282252-kube-api-access-5w7nh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p\" (UID: \"df7cc2c0-554d-4cb6-a830-27d7d0282252\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.600274 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.600250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:41.731246 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:41.731215 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p"] Jul 15 10:30:41.733483 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:30:41.733458 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf7cc2c0_554d_4cb6_a830_27d7d0282252.slice/crio-d3d57a8847cd278dfcc7d24fd323f2315bbc946de71eb3b59327f6d18863e91a WatchSource:0}: Error finding container d3d57a8847cd278dfcc7d24fd323f2315bbc946de71eb3b59327f6d18863e91a: Status 404 returned error can't find the container with id d3d57a8847cd278dfcc7d24fd323f2315bbc946de71eb3b59327f6d18863e91a Jul 15 10:30:42.142989 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:42.142955 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" event={"ID":"df7cc2c0-554d-4cb6-a830-27d7d0282252","Type":"ContainerStarted","Data":"e006d7351e29fa147c8b66b8b2f8a36a5452befaf5c58064c1af829a571e5153"} Jul 15 10:30:42.143126 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:42.143003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" event={"ID":"df7cc2c0-554d-4cb6-a830-27d7d0282252","Type":"ContainerStarted","Data":"d3d57a8847cd278dfcc7d24fd323f2315bbc946de71eb3b59327f6d18863e91a"} Jul 15 10:30:42.165874 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:42.165829 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" podStartSLOduration=0.948214868 podStartE2EDuration="1.165814998s" totalImagesPullingTime="217.60013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:30:42.163764857 +0000 UTC m=+644.296958196" watchObservedRunningTime="2026-07-15 10:30:42.165814998 +0000 UTC m=+644.299008337" Jul 15 10:30:44.210835 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:44.210797 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:44.211248 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:44.210889 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:44.223067 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:44.223045 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:45.175373 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:45.175340 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gjm86" Jul 15 10:30:51.600438 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:51.600399 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:51.600852 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:51.600453 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:51.612644 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:51.612615 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:30:52.210011 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:30:52.209982 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p" Jul 15 10:31:11.147929 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:11.147893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:31:22.354578 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:22.354546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:31:56.122014 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.121924 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bc45ddbc6-2hlk6"] Jul 15 10:31:56.125491 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.125469 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" Jul 15 10:31:56.138355 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.138333 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bc45ddbc6-2hlk6"] Jul 15 10:31:56.242672 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.242650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb119164-c78d-4fc7-a093-bf9508685d95-tls-cert\") pod \"authorino-bc45ddbc6-2hlk6\" (UID: \"cb119164-c78d-4fc7-a093-bf9508685d95\") " pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" Jul 15 10:31:56.242789 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.242713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5js9\" (UniqueName: \"kubernetes.io/projected/cb119164-c78d-4fc7-a093-bf9508685d95-kube-api-access-f5js9\") pod \"authorino-bc45ddbc6-2hlk6\" (UID: \"cb119164-c78d-4fc7-a093-bf9508685d95\") " pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" Jul 15 10:31:56.343974 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.343952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5js9\" (UniqueName: \"kubernetes.io/projected/cb119164-c78d-4fc7-a093-bf9508685d95-kube-api-access-f5js9\") pod \"authorino-bc45ddbc6-2hlk6\" (UID: \"cb119164-c78d-4fc7-a093-bf9508685d95\") " pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" Jul 15 10:31:56.344073 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.343992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb119164-c78d-4fc7-a093-bf9508685d95-tls-cert\") pod \"authorino-bc45ddbc6-2hlk6\" (UID: \"cb119164-c78d-4fc7-a093-bf9508685d95\") " pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" Jul 15 10:31:56.346655 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.346631 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb119164-c78d-4fc7-a093-bf9508685d95-tls-cert\") pod \"authorino-bc45ddbc6-2hlk6\" (UID: \"cb119164-c78d-4fc7-a093-bf9508685d95\") " pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" Jul 15 10:31:56.354837 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.354810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5js9\" (UniqueName: \"kubernetes.io/projected/cb119164-c78d-4fc7-a093-bf9508685d95-kube-api-access-f5js9\") pod \"authorino-bc45ddbc6-2hlk6\" (UID: \"cb119164-c78d-4fc7-a093-bf9508685d95\") " pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" Jul 15 10:31:56.376239 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.376188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:31:56.433915 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.433888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" Jul 15 10:31:56.476931 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.476896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:31:56.572099 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:56.571921 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bc45ddbc6-2hlk6"] Jul 15 10:31:56.574752 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:31:56.574703 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb119164_c78d_4fc7_a093_bf9508685d95.slice/crio-8d1d2048cddffff02cbd31d18718856e7ff417d77599fd1bc03411e23de4c30c WatchSource:0}: Error finding container 8d1d2048cddffff02cbd31d18718856e7ff417d77599fd1bc03411e23de4c30c: Status 404 returned error can't find the container with id 8d1d2048cddffff02cbd31d18718856e7ff417d77599fd1bc03411e23de4c30c Jul 15 10:31:57.500169 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.500134 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" event={"ID":"cb119164-c78d-4fc7-a093-bf9508685d95","Type":"ContainerStarted","Data":"0af425cbaacdc402d943968f7fe3b73bb672e61a4a8810c27fbf6fb7fa63d2e1"} Jul 15 10:31:57.500551 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.500185 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" event={"ID":"cb119164-c78d-4fc7-a093-bf9508685d95","Type":"ContainerStarted","Data":"8d1d2048cddffff02cbd31d18718856e7ff417d77599fd1bc03411e23de4c30c"} Jul 15 10:31:57.526377 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.526315 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bc45ddbc6-2hlk6" podStartSLOduration=1.077030633 podStartE2EDuration="1.526292867s" totalImagesPullingTime="449.262234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:31:57.525253889 +0000 UTC m=+719.658447232" watchObservedRunningTime="2026-07-15 10:31:57.526292867 +0000 UTC m=+719.659486206" Jul 15 10:31:57.543110 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.543074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:31:57.562587 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.562552 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77456d549c-d8jr4"] Jul 15 10:31:57.562824 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.562798 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-77456d549c-d8jr4" podUID="cfa8f7b0-0c86-45f0-808f-6d1807a7c75b" containerName="authorino" containerID="cri-o://354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b" gracePeriod=30 Jul 15 10:31:57.622578 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.622538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:31:57.832583 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.832557 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77456d549c-d8jr4" Jul 15 10:31:57.957688 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.957660 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twxt9\" (UniqueName: \"kubernetes.io/projected/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-kube-api-access-twxt9\") pod \"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b\" (UID: \"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b\") " Jul 15 10:31:57.957875 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.957712 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-tls-cert\") pod \"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b\" (UID: \"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b\") " Jul 15 10:31:57.959962 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.959929 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-kube-api-access-twxt9" (OuterVolumeSpecName: "kube-api-access-twxt9") pod "cfa8f7b0-0c86-45f0-808f-6d1807a7c75b" (UID: "cfa8f7b0-0c86-45f0-808f-6d1807a7c75b"). InnerVolumeSpecName "kube-api-access-twxt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:31:57.968633 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:57.968611 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "cfa8f7b0-0c86-45f0-808f-6d1807a7c75b" (UID: "cfa8f7b0-0c86-45f0-808f-6d1807a7c75b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 10:31:58.059015 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.058918 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twxt9\" (UniqueName: \"kubernetes.io/projected/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-kube-api-access-twxt9\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:31:58.059015 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.058959 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b-tls-cert\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:31:58.506863 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.506819 2579 generic.go:356] "Generic (PLEG): container finished" podID="cfa8f7b0-0c86-45f0-808f-6d1807a7c75b" containerID="354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b" exitCode=0 Jul 15 10:31:58.507330 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.506865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77456d549c-d8jr4" event={"ID":"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b","Type":"ContainerDied","Data":"354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b"} Jul 15 10:31:58.507330 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.506885 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77456d549c-d8jr4" Jul 15 10:31:58.507330 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.506930 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77456d549c-d8jr4" event={"ID":"cfa8f7b0-0c86-45f0-808f-6d1807a7c75b","Type":"ContainerDied","Data":"57bb1b947c1117fa87d55b074a0f9b94ef84122ebdd98be0c2199447e8889e87"} Jul 15 10:31:58.507330 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.506963 2579 scope.go:117] "RemoveContainer" containerID="354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b" Jul 15 10:31:58.516841 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.516810 2579 scope.go:117] "RemoveContainer" containerID="354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b" Jul 15 10:31:58.517204 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:31:58.517176 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b\": container with ID starting with 354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b not found: ID does not exist" containerID="354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b" Jul 15 10:31:58.517320 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.517208 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b"} err="failed to get container status \"354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b\": rpc error: code = NotFound desc = could not find container \"354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b\": container with ID starting with 354ac8c24e5009d4bcac5c267e60ef6bf797034474aee6a06913bf8ba4ebd99b not found: ID does not exist" Jul 15 10:31:58.536364 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.536337 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77456d549c-d8jr4"] Jul 15 10:31:58.541398 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:31:58.541377 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-77456d549c-d8jr4"] Jul 15 10:32:00.351917 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:32:00.351885 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa8f7b0-0c86-45f0-808f-6d1807a7c75b" path="/var/lib/kubelet/pods/cfa8f7b0-0c86-45f0-808f-6d1807a7c75b/volumes" Jul 15 10:32:46.146393 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:32:46.146357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:33:01.457278 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:33:01.457239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:33:01.561056 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:33:01.561015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:33:18.270231 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:33:18.270198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:33:19.722256 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:33:19.722216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:33:58.728522 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:33:58.728484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:33:58.811157 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:33:58.811123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:33:58.950887 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:33:58.950862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:34:07.706707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:34:07.706674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:34:07.808585 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:34:07.808561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:34:07.903457 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:34:07.903427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:34:08.097432 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:34:08.097407 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:34:14.456665 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:34:14.456628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:34:14.562899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:34:14.562872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:34:58.298849 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:34:58.298815 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:34:58.299345 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:34:58.299277 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:35:17.494447 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:35:17.494398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:35:18.859727 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:35:18.859694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:35:44.535806 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:35:44.535769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:35:44.634816 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:35:44.634783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:36:15.896942 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:36:15.896893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:36:54.691405 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:36:54.691370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:39:41.048840 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:39:41.048806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:39:41.118789 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:39:41.118761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:39:58.324706 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:39:58.324677 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:39:58.327467 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:39:58.327444 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:43:39.539551 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:43:39.539516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:44:58.354901 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:44:58.354874 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:44:58.359077 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:44:58.359050 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:44:59.962877 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:44:59.962835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:00.057086 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:00.057047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:06.321400 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.321365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:06.382757 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.382721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:06.513180 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.513156 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2"] Jul 15 10:45:06.513496 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.513483 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa8f7b0-0c86-45f0-808f-6d1807a7c75b" containerName="authorino" Jul 15 10:45:06.513548 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.513497 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa8f7b0-0c86-45f0-808f-6d1807a7c75b" containerName="authorino" Jul 15 10:45:06.513588 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.513555 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa8f7b0-0c86-45f0-808f-6d1807a7c75b" containerName="authorino" Jul 15 10:45:06.516486 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.516464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.519949 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.519928 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5b874627-openshift-default-dockercfg-k7rqp\"" Jul 15 10:45:06.539097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.539071 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2"] Jul 15 10:45:06.546363 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.546337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:06.633411 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.633385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:06.697806 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.697780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-credential-socket\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.697911 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.697818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istiod-ca-cert\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.697911 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.697876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5b9\" (UniqueName: \"kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-kube-api-access-2d5b9\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.697911 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.697904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-certs\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.698027 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.697973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-podinfo\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.698027 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.698009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-socket\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.698097 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.698062 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-token\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.698186 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.698107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-envoy\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.698186 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.698163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-data\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.799712 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.799687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-credential-socket\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.799817 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.799730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istiod-ca-cert\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.799817 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.799788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5b9\" (UniqueName: \"kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-kube-api-access-2d5b9\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.799915 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.799815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-certs\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.799915 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.799851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-podinfo\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.799915 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.799890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-socket\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.800054 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.799935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-token\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.800054 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.799994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-envoy\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.800201 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.800179 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-credential-socket\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.800281 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.800192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-certs\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.800456 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.800438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istiod-ca-cert\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.800544 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.800525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-data\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.800666 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.800645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-socket\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.800983 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.800878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-data\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.802576 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.802554 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-podinfo\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.802703 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.802688 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-envoy\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.810349 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.810326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-token\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.810990 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.810967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5b9\" (UniqueName: \"kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-kube-api-access-2d5b9\") pod \"e2e-ait-5b874627-openshift-default-589df85b7c-jjls2\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.827402 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.827382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:06.970313 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.970282 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2"] Jul 15 10:45:06.972780 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:45:06.972732 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd8a4a8_7cdb_4c70_9e8f_6876f5c1be83.slice/crio-b811dd028989eaa9eeb1e37766ca725f1d7152e645f19da4ae6af2cb298d31d0 WatchSource:0}: Error finding container b811dd028989eaa9eeb1e37766ca725f1d7152e645f19da4ae6af2cb298d31d0: Status 404 returned error can't find the container with id b811dd028989eaa9eeb1e37766ca725f1d7152e645f19da4ae6af2cb298d31d0 Jul 15 10:45:06.975486 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.975441 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:45:06.975590 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.975538 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:45:06.975590 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:06.975584 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:45:07.199727 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:07.199648 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" event={"ID":"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83","Type":"ContainerStarted","Data":"3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84"} Jul 15 10:45:07.199727 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:07.199703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" event={"ID":"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83","Type":"ContainerStarted","Data":"b811dd028989eaa9eeb1e37766ca725f1d7152e645f19da4ae6af2cb298d31d0"} Jul 15 10:45:07.236003 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:07.235944 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" podStartSLOduration=1.235924093 podStartE2EDuration="1.235924093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:07.23447126 +0000 UTC m=+1509.367664598" watchObservedRunningTime="2026-07-15 10:45:07.235924093 +0000 UTC m=+1509.369117432" Jul 15 10:45:07.572599 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:07.572518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:07.696899 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:07.696838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:07.791200 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:07.791170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:07.828188 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:07.828128 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:07.893798 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:07.893775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:08.018351 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:08.018316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:08.580931 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:08.580892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:08.681466 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:08.681441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:08.833174 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:08.833103 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:09.209390 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:09.209363 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:09.215910 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:09.215881 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:10.236412 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:10.236382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:45:18.252120 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:18.252060 2579 generic.go:356] "Generic (PLEG): container finished" podID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerID="3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84" exitCode=0 Jul 15 10:45:18.252480 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:18.252133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" event={"ID":"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83","Type":"ContainerDied","Data":"3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84"} Jul 15 10:45:18.252547 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:18.252519 2579 scope.go:117] "RemoveContainer" containerID="3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84" Jul 15 10:45:18.253469 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:18.253438 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:45:18.253560 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:18.253504 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:45:18.253560 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:18.253537 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:45:19.258749 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:19.258705 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" event={"ID":"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83","Type":"ContainerStarted","Data":"866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7"} Jul 15 10:45:19.828303 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:19.828261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:19.829364 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:19.829339 2579 patch_prober.go:28] interesting pod/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 15 10:45:19.829421 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:19.829390 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 15 10:45:20.828334 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:20.828298 2579 patch_prober.go:28] interesting pod/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 15 10:45:20.828730 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:20.828354 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 15 10:45:21.828614 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:21.828578 2579 patch_prober.go:28] interesting pod/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 15 10:45:21.829010 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:21.828629 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 15 10:45:22.828313 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:22.828281 2579 patch_prober.go:28] interesting pod/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 15 10:45:22.828495 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:22.828335 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 15 10:45:23.832163 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:23.832125 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:23.832617 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:23.832377 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:45:23.833088 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:45:23.833064 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:46:13.395388 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:13.395353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:18.042374 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:18.042335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:25.065470 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:25.065426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:25.204614 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:25.204583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:28.157530 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:28.157493 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2"] Jul 15 10:46:28.157935 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:28.157718 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" containerID="cri-o://866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7" gracePeriod=30 Jul 15 10:46:28.186399 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:28.186367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:28.579369 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:28.579284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:28.765292 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:28.765261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:28.846071 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:28.846045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:30.034226 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:30.034192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:30.169025 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:30.168995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:30.262315 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:30.262291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:30.753264 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:30.753236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:30.875968 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:30.875928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:31.636253 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:31.636219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:33.404771 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.404726 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:46:33.520421 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520390 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-token\") pod \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " Jul 15 10:46:33.520571 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520453 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istiod-ca-cert\") pod \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " Jul 15 10:46:33.520571 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520476 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-certs\") pod \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " Jul 15 10:46:33.520689 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520600 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-data\") pod \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " Jul 15 10:46:33.520689 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520646 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-socket\") pod \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " Jul 15 10:46:33.520689 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520675 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-envoy\") pod \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " Jul 15 10:46:33.520870 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520701 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" (UID: "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:46:33.520870 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-credential-socket\") pod \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " Jul 15 10:46:33.520978 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520867 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-podinfo\") pod \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " Jul 15 10:46:33.520978 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520914 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d5b9\" (UniqueName: \"kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-kube-api-access-2d5b9\") pod \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\" (UID: \"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83\") " Jul 15 10:46:33.520978 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520947 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" (UID: "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:46:33.521113 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.520985 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" (UID: "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 10:46:33.521210 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.521189 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istiod-ca-cert\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:46:33.521285 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.521214 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-certs\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:46:33.521285 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.521227 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-workload-socket\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:46:33.521384 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.521266 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" (UID: "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:46:33.521440 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.521381 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-data" (OuterVolumeSpecName: "istio-data") pod "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" (UID: "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:46:33.523224 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.523198 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" (UID: "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 10:46:33.523321 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.523220 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-kube-api-access-2d5b9" (OuterVolumeSpecName: "kube-api-access-2d5b9") pod "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" (UID: "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83"). InnerVolumeSpecName "kube-api-access-2d5b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:46:33.523362 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.523312 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-token" (OuterVolumeSpecName: "istio-token") pod "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" (UID: "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:46:33.523477 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.523463 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" (UID: "ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:46:33.604425 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.604387 2579 generic.go:356] "Generic (PLEG): container finished" podID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerID="866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7" exitCode=0 Jul 15 10:46:33.604589 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.604439 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" Jul 15 10:46:33.604589 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.604474 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" event={"ID":"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83","Type":"ContainerDied","Data":"866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7"} Jul 15 10:46:33.604589 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.604549 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2" event={"ID":"ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83","Type":"ContainerDied","Data":"b811dd028989eaa9eeb1e37766ca725f1d7152e645f19da4ae6af2cb298d31d0"} Jul 15 10:46:33.604589 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.604571 2579 scope.go:117] "RemoveContainer" containerID="866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7" Jul 15 10:46:33.614379 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.614363 2579 scope.go:117] "RemoveContainer" containerID="3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84" Jul 15 10:46:33.622395 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.622378 2579 scope.go:117] "RemoveContainer" containerID="866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7" Jul 15 10:46:33.622562 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.622545 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-podinfo\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:46:33.622624 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.622571 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d5b9\" (UniqueName: \"kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-kube-api-access-2d5b9\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:46:33.622624 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.622588 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-token\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:46:33.622624 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.622602 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-data\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:46:33.622624 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.622615 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-istio-envoy\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:46:33.622786 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.622627 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83-credential-socket\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:46:33.622786 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:46:33.622651 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7\": container with ID starting with 866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7 not found: ID does not exist" containerID="866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7" Jul 15 10:46:33.622786 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.622675 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7"} err="failed to get container status \"866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7\": rpc error: code = NotFound desc = could not find container \"866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7\": container with ID starting with 866e7d7cee2a400272db4432c30f9a54e50b75526638eefdd5e07384d39283c7 not found: ID does not exist" Jul 15 10:46:33.622786 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.622692 2579 scope.go:117] "RemoveContainer" containerID="3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84" Jul 15 10:46:33.622961 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:46:33.622942 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84\": container with ID starting with 3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84 not found: ID does not exist" containerID="3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84" Jul 15 10:46:33.622998 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.622967 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84"} err="failed to get container status \"3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84\": rpc error: code = NotFound desc = could not find container \"3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84\": container with ID starting with 3dcccab8187a7ec7a5af1f4760c0d4eebc8dd42b93535dcea3e215d8fe6c7a84 not found: ID does not exist" Jul 15 10:46:33.630139 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.630112 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2"] Jul 15 10:46:33.633004 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:33.632982 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5b874627-openshift-default-589df85b7c-jjls2"] Jul 15 10:46:34.351355 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:34.351320 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" path="/var/lib/kubelet/pods/ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83/volumes" Jul 15 10:46:35.098875 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:35.098830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:35.199157 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:35.199096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:38.537072 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:46:38.537041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:34.695443 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:34.695383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:34.788558 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:34.788509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:41.205080 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:41.205049 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:41.283655 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:41.283620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:42.549923 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:42.549893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:49.510413 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.510372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:49.762973 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.762912 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw"] Jul 15 10:47:49.763228 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.763217 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" Jul 15 10:47:49.763269 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.763230 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" Jul 15 10:47:49.763309 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.763299 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" Jul 15 10:47:49.766382 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.766366 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.769754 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.769712 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8452aa9e-openshift-default-dockercfg-9knm4\"" Jul 15 10:47:49.785667 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.785646 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw"] Jul 15 10:47:49.868439 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.868406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-certs\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.868562 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.868442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-envoy\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.868562 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.868475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26290275-1be4-4492-9752-eb83bb6acf26-istiod-ca-cert\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.868638 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.868583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26290275-1be4-4492-9752-eb83bb6acf26-istio-podinfo\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.868638 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.868614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9gh\" (UniqueName: \"kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-kube-api-access-qs9gh\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.868700 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.868688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-data\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.868768 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.868717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-istio-token\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.868825 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.868776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-socket\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.868825 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.868795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-credential-socket\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.969542 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.969519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26290275-1be4-4492-9752-eb83bb6acf26-istiod-ca-cert\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.969662 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.969567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26290275-1be4-4492-9752-eb83bb6acf26-istio-podinfo\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.969662 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.969586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9gh\" (UniqueName: \"kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-kube-api-access-qs9gh\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.969662 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.969618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-data\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.969858 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.969776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-istio-token\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.969858 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.969827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-socket\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.970010 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.969857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-credential-socket\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.970010 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.969948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-certs\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.970010 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.969966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-envoy\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.970159 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.970050 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-data\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.970159 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.970110 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-socket\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.970369 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.970350 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-credential-socket\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.970445 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.970387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-certs\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.970445 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.970415 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26290275-1be4-4492-9752-eb83bb6acf26-istiod-ca-cert\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.972253 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.972232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26290275-1be4-4492-9752-eb83bb6acf26-istio-podinfo\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.972346 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.972261 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-envoy\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.984018 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.983993 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-istio-token\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:49.988037 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:49.988016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9gh\" (UniqueName: \"kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-kube-api-access-qs9gh\") pod \"e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:50.076627 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.076578 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:50.217589 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.217564 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw"] Jul 15 10:47:50.219517 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:47:50.219494 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26290275_1be4_4492_9752_eb83bb6acf26.slice/crio-dd3e1d8b5f3af7818e2fd83622e761fdc58d19719eb81fe4bbd206c22be4acc1 WatchSource:0}: Error finding container dd3e1d8b5f3af7818e2fd83622e761fdc58d19719eb81fe4bbd206c22be4acc1: Status 404 returned error can't find the container with id dd3e1d8b5f3af7818e2fd83622e761fdc58d19719eb81fe4bbd206c22be4acc1 Jul 15 10:47:50.221550 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.221518 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:47:50.221644 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.221605 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:47:50.221705 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.221648 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:47:50.224496 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.224472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:50.289215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.289182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:50.944673 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.944635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:50.960164 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.960136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" event={"ID":"26290275-1be4-4492-9752-eb83bb6acf26","Type":"ContainerStarted","Data":"01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c"} Jul 15 10:47:50.960315 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.960181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" event={"ID":"26290275-1be4-4492-9752-eb83bb6acf26","Type":"ContainerStarted","Data":"dd3e1d8b5f3af7818e2fd83622e761fdc58d19719eb81fe4bbd206c22be4acc1"} Jul 15 10:47:50.997912 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:50.997865 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" podStartSLOduration=1.997850913 podStartE2EDuration="1.997850913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:50.996509544 +0000 UTC m=+1673.129702880" watchObservedRunningTime="2026-07-15 10:47:50.997850913 +0000 UTC m=+1673.131044243" Jul 15 10:47:51.052796 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:51.052725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:51.076800 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:51.076776 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:51.081829 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:51.081804 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:51.772815 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:51.772771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:51.868887 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:51.868842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:51.964683 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:51.964646 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:51.965621 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:51.965602 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:47:53.145520 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:47:53.145488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:02.017637 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:02.017591 2579 generic.go:356] "Generic (PLEG): container finished" podID="26290275-1be4-4492-9752-eb83bb6acf26" containerID="01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c" exitCode=0 Jul 15 10:48:02.018088 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:02.017664 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" event={"ID":"26290275-1be4-4492-9752-eb83bb6acf26","Type":"ContainerDied","Data":"01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c"} Jul 15 10:48:02.018088 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:02.018062 2579 scope.go:117] "RemoveContainer" containerID="01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c" Jul 15 10:48:02.019019 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:02.018988 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:48:02.019089 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:02.019049 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:48:02.019089 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:02.019081 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:48:03.024185 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:03.024151 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" event={"ID":"26290275-1be4-4492-9752-eb83bb6acf26","Type":"ContainerStarted","Data":"f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad"} Jul 15 10:48:03.077304 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:03.077277 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:48:03.078395 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:03.078373 2579 patch_prober.go:28] interesting pod/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 15 10:48:03.078451 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:03.078431 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 15 10:48:04.077137 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:04.077103 2579 patch_prober.go:28] interesting pod/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 15 10:48:04.077513 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:04.077153 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 15 10:48:05.077071 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:05.077030 2579 patch_prober.go:28] interesting pod/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 15 10:48:05.077310 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:05.077103 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 15 10:48:06.077027 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:06.076993 2579 patch_prober.go:28] interesting pod/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 15 10:48:06.077286 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:06.077047 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 15 10:48:06.965488 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:06.965451 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:48:07.080888 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:07.080858 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:48:07.081809 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:07.081789 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:48:32.422452 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:32.422364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:32.514949 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:32.514920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:41.843543 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:41.843510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:44.014721 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:44.014682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:57.014925 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:57.014890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:57.108780 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:48:57.108754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:06.909827 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:06.909794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:11.256781 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:11.256711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:19.289874 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:19.289832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:19.386943 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:19.386911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:22.385156 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:22.385114 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw"] Jul 15 10:49:22.385672 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:22.385320 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" containerID="cri-o://f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad" gracePeriod=30 Jul 15 10:49:22.400380 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:22.400344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:22.749165 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:22.749133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:23.870188 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:23.870154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:23.968851 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:23.968776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:24.068240 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:24.068208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:24.328198 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:24.328128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:24.752390 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:24.752361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:24.852149 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:24.852105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:27.621209 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.621185 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:49:27.777734 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.777706 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-certs\") pod \"26290275-1be4-4492-9752-eb83bb6acf26\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " Jul 15 10:49:27.777898 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.777758 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-socket\") pod \"26290275-1be4-4492-9752-eb83bb6acf26\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " Jul 15 10:49:27.777898 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.777778 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26290275-1be4-4492-9752-eb83bb6acf26-istio-podinfo\") pod \"26290275-1be4-4492-9752-eb83bb6acf26\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " Jul 15 10:49:27.777898 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.777820 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs9gh\" (UniqueName: \"kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-kube-api-access-qs9gh\") pod \"26290275-1be4-4492-9752-eb83bb6acf26\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " Jul 15 10:49:27.777898 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.777837 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-credential-socket\") pod \"26290275-1be4-4492-9752-eb83bb6acf26\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " Jul 15 10:49:27.777898 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.777860 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-data\") pod \"26290275-1be4-4492-9752-eb83bb6acf26\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " Jul 15 10:49:27.777898 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.777891 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-istio-token\") pod \"26290275-1be4-4492-9752-eb83bb6acf26\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " Jul 15 10:49:27.778401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.777956 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26290275-1be4-4492-9752-eb83bb6acf26-istiod-ca-cert\") pod \"26290275-1be4-4492-9752-eb83bb6acf26\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " Jul 15 10:49:27.778401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.777960 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26290275-1be4-4492-9752-eb83bb6acf26" (UID: "26290275-1be4-4492-9752-eb83bb6acf26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:49:27.778401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.778028 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26290275-1be4-4492-9752-eb83bb6acf26" (UID: "26290275-1be4-4492-9752-eb83bb6acf26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:49:27.778401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.778038 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-envoy\") pod \"26290275-1be4-4492-9752-eb83bb6acf26\" (UID: \"26290275-1be4-4492-9752-eb83bb6acf26\") " Jul 15 10:49:27.778401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.778246 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26290275-1be4-4492-9752-eb83bb6acf26" (UID: "26290275-1be4-4492-9752-eb83bb6acf26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:49:27.778401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.778369 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-certs\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:49:27.778401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.778382 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-workload-socket\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:49:27.778401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.778394 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-credential-socket\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:49:27.778831 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.778656 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-data" (OuterVolumeSpecName: "istio-data") pod "26290275-1be4-4492-9752-eb83bb6acf26" (UID: "26290275-1be4-4492-9752-eb83bb6acf26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:49:27.779003 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.778978 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26290275-1be4-4492-9752-eb83bb6acf26-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26290275-1be4-4492-9752-eb83bb6acf26" (UID: "26290275-1be4-4492-9752-eb83bb6acf26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 10:49:27.780126 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.780096 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26290275-1be4-4492-9752-eb83bb6acf26-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26290275-1be4-4492-9752-eb83bb6acf26" (UID: "26290275-1be4-4492-9752-eb83bb6acf26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 10:49:27.780468 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.780446 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-istio-token" (OuterVolumeSpecName: "istio-token") pod "26290275-1be4-4492-9752-eb83bb6acf26" (UID: "26290275-1be4-4492-9752-eb83bb6acf26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:49:27.780734 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.780717 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26290275-1be4-4492-9752-eb83bb6acf26" (UID: "26290275-1be4-4492-9752-eb83bb6acf26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:49:27.780734 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.780757 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-kube-api-access-qs9gh" (OuterVolumeSpecName: "kube-api-access-qs9gh") pod "26290275-1be4-4492-9752-eb83bb6acf26" (UID: "26290275-1be4-4492-9752-eb83bb6acf26"). InnerVolumeSpecName "kube-api-access-qs9gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:49:27.878978 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.878947 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-istio-token\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:49:27.878978 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.878977 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26290275-1be4-4492-9752-eb83bb6acf26-istiod-ca-cert\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:49:27.878978 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.878985 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-envoy\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:49:27.879201 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.878994 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26290275-1be4-4492-9752-eb83bb6acf26-istio-podinfo\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:49:27.879201 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.879002 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs9gh\" (UniqueName: \"kubernetes.io/projected/26290275-1be4-4492-9752-eb83bb6acf26-kube-api-access-qs9gh\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:49:27.879201 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:27.879011 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26290275-1be4-4492-9752-eb83bb6acf26-istio-data\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:49:28.431657 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.431630 2579 generic.go:356] "Generic (PLEG): container finished" podID="26290275-1be4-4492-9752-eb83bb6acf26" containerID="f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad" exitCode=0 Jul 15 10:49:28.431837 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.431682 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" event={"ID":"26290275-1be4-4492-9752-eb83bb6acf26","Type":"ContainerDied","Data":"f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad"} Jul 15 10:49:28.431837 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.431690 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" Jul 15 10:49:28.431837 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.431720 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw" event={"ID":"26290275-1be4-4492-9752-eb83bb6acf26","Type":"ContainerDied","Data":"dd3e1d8b5f3af7818e2fd83622e761fdc58d19719eb81fe4bbd206c22be4acc1"} Jul 15 10:49:28.431837 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.431768 2579 scope.go:117] "RemoveContainer" containerID="f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad" Jul 15 10:49:28.443342 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.443321 2579 scope.go:117] "RemoveContainer" containerID="01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c" Jul 15 10:49:28.450982 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.450965 2579 scope.go:117] "RemoveContainer" containerID="f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad" Jul 15 10:49:28.451219 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:49:28.451201 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad\": container with ID starting with f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad not found: ID does not exist" containerID="f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad" Jul 15 10:49:28.451261 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.451227 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad"} err="failed to get container status \"f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad\": rpc error: code = NotFound desc = could not find container \"f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad\": container with ID starting with f8022d58e8bfd2676d53a04b3559fa1f09022e5ec0eb6f94b12af804086636ad not found: ID does not exist" Jul 15 10:49:28.451261 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.451248 2579 scope.go:117] "RemoveContainer" containerID="01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c" Jul 15 10:49:28.451462 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:49:28.451439 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c\": container with ID starting with 01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c not found: ID does not exist" containerID="01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c" Jul 15 10:49:28.451502 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.451469 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c"} err="failed to get container status \"01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c\": rpc error: code = NotFound desc = could not find container \"01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c\": container with ID starting with 01904971f1e2e2ca40a12e91ec5e4d1d6cc1b1e5212fec73b8edbb76cf21f07c not found: ID does not exist" Jul 15 10:49:28.464276 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.464256 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw"] Jul 15 10:49:28.465864 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:28.465841 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8452aa9e-openshift-default-5765ff457c-w9rcw"] Jul 15 10:49:30.351534 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:30.351496 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26290275-1be4-4492-9752-eb83bb6acf26" path="/var/lib/kubelet/pods/26290275-1be4-4492-9752-eb83bb6acf26/volumes" Jul 15 10:49:34.602154 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:34.602113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:34.704143 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:34.704112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:37.677991 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:37.677959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:58.385577 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:58.385542 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:49:58.390830 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:49:58.390809 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:50:10.655311 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:10.655276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:10.756430 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:10.756401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:39.177495 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:39.177455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:43.486124 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:43.486086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:50.736045 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:50.736012 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:50.803805 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:50.803771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:53.850080 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:53.850043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:54.329286 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:54.329211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:54.427679 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:54.427646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:54.530900 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:54.530871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:56.040330 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:50:56.040295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:00.191886 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:00.191850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:02.771302 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:02.771268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:02.878470 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:02.878430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:02.976843 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:02.976809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:03.076818 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:03.076683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:03.520447 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:03.520412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:29.594323 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:29.594235 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:36.621574 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:36.621534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:36.693401 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:36.693370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:36.781586 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:36.781555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:36.889870 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:36.889779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:52.850906 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:52.850869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:59.649929 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:59.649894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:59.753244 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:59.753209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:59.853793 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:51:59.853761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:10.445199 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:10.445162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:11.313503 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:11.313463 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:18.640303 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:18.640262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:18.744198 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:18.744166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:18.844563 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:18.844531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:18.942890 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:18.942822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:19.751969 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:19.751940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:19.851526 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:19.851495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:19.952081 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:19.952057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:21.699697 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:21.699667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:29.021294 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:29.021262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:29.082948 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:29.082917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:29.771089 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:29.771054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:36.468252 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:36.468213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:36.567897 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:36.567867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:36.744536 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:36.744451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:37.095560 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:37.095534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:37.341602 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:37.341568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:40.351963 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:40.351926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:40.434919 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:40.434891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:40.506793 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:40.506762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:40.673449 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:40.673361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:41.086087 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:41.086002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:41.186699 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:41.186667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:44.908616 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:44.908578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:45.612581 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:45.612536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:45.718074 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:45.718044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:46.048302 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:46.048216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:52.477145 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:52.477065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:52.555394 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:52:52.555359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:03.116326 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:03.116297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:07.960570 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:07.960533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:14.379907 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:14.379868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:14.480087 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:14.480052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:18.679685 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:18.679644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:18.893551 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:18.893517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:19.032949 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:19.032872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:19.126515 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:19.126486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:19.248216 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:19.248189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:25.817147 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:25.817110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:25.878624 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:25.878585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:25.958194 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:25.958158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:26.057988 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:26.057948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:26.195789 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:26.195758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:28.867460 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:28.867430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:32.528442 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:32.528390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:32.629157 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:32.629114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:36.072681 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:36.072642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:39.354693 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:39.354659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:39.684564 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:39.684473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:39.832060 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:39.832008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:42.612052 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:42.612017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:42.691779 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:42.691727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:43.097941 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:43.097907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:43.197614 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:43.197590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:43.512680 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:43.512601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:43.761430 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:43.761395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:44.017102 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:44.017073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:45.723982 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:45.723941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:47.120296 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:53:47.120258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:01.285690 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.285649 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689"] Jul 15 10:54:01.286179 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.286074 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" Jul 15 10:54:01.286179 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.286088 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" Jul 15 10:54:01.286179 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.286101 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" Jul 15 10:54:01.286179 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.286107 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" Jul 15 10:54:01.286179 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.286117 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" Jul 15 10:54:01.286179 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.286123 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" Jul 15 10:54:01.286179 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.286179 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" Jul 15 10:54:01.286424 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.286189 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd8a4a8-7cdb-4c70-9e8f-6876f5c1be83" containerName="istio-proxy" Jul 15 10:54:01.289169 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.289153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.292501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.292475 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e595ab4b\"/\"e2e-lifecycle-model-e595ab4b-kserve-self-signed-certs\"" Jul 15 10:54:01.292640 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.292475 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e595ab4b\"/\"default-dockercfg-tmfds\"" Jul 15 10:54:01.292640 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.292481 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e595ab4b\"/\"openshift-service-ca.crt\"" Jul 15 10:54:01.292640 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.292508 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e595ab4b\"/\"kube-root-ca.crt\"" Jul 15 10:54:01.299605 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.299579 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689"] Jul 15 10:54:01.321448 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.321428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-home\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.321557 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.321455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-model-cache\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.321557 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.321485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-tmp-dir\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.321557 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.321546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-dshm\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.321715 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.321572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17df043-eb89-468a-9fd0-c19e55353970-tls-certs\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.321715 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.321632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jqg2\" (UniqueName: \"kubernetes.io/projected/a17df043-eb89-468a-9fd0-c19e55353970-kube-api-access-5jqg2\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.390201 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.390172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:01.422215 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.422191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-tmp-dir\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.422317 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.422234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-dshm\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.422361 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.422345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17df043-eb89-468a-9fd0-c19e55353970-tls-certs\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.422403 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.422389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jqg2\" (UniqueName: \"kubernetes.io/projected/a17df043-eb89-468a-9fd0-c19e55353970-kube-api-access-5jqg2\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.422529 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.422511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-home\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.422598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.422548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-model-cache\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.422598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.422586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-tmp-dir\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.422876 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.422851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-home\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.422971 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.422875 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-model-cache\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.424509 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.424491 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-dshm\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.424875 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.424854 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17df043-eb89-468a-9fd0-c19e55353970-tls-certs\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.434554 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.434526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jqg2\" (UniqueName: \"kubernetes.io/projected/a17df043-eb89-468a-9fd0-c19e55353970-kube-api-access-5jqg2\") pod \"e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.599309 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.599281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:01.734504 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:01.734469 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689"] Jul 15 10:54:01.737292 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:54:01.737268 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda17df043_eb89_468a_9fd0_c19e55353970.slice/crio-dbf4ee8285e2915e0d17fd424041b09491f939e1882c6f1cad5856fddf01cea6 WatchSource:0}: Error finding container dbf4ee8285e2915e0d17fd424041b09491f939e1882c6f1cad5856fddf01cea6: Status 404 returned error can't find the container with id dbf4ee8285e2915e0d17fd424041b09491f939e1882c6f1cad5856fddf01cea6 Jul 15 10:54:02.696873 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:02.696844 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" event={"ID":"a17df043-eb89-468a-9fd0-c19e55353970","Type":"ContainerStarted","Data":"0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6"} Jul 15 10:54:02.697246 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:02.696883 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" event={"ID":"a17df043-eb89-468a-9fd0-c19e55353970","Type":"ContainerStarted","Data":"dbf4ee8285e2915e0d17fd424041b09491f939e1882c6f1cad5856fddf01cea6"} Jul 15 10:54:11.599993 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:11.599958 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:11.600490 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:11.600015 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:11.612800 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:11.612778 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:11.633136 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:11.633090 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" podStartSLOduration=10.633078526 podStartE2EDuration="10.633078526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:54:02.715353492 +0000 UTC m=+2044.848546842" watchObservedRunningTime="2026-07-15 10:54:11.633078526 +0000 UTC m=+2053.766271864" Jul 15 10:54:11.751819 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:11.751800 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:31.617351 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:31.617274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:36.060022 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:36.059989 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:41.053870 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:41.053835 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689"] Jul 15 10:54:41.146708 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:41.146671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:56.074246 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.074183 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" podUID="a17df043-eb89-468a-9fd0-c19e55353970" containerName="main" containerID="cri-o://0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6" gracePeriod=45 Jul 15 10:54:56.312647 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.312621 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:56.342762 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.342701 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-tmp-dir\") pod \"a17df043-eb89-468a-9fd0-c19e55353970\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " Jul 15 10:54:56.342867 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.342811 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-home\") pod \"a17df043-eb89-468a-9fd0-c19e55353970\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " Jul 15 10:54:56.342867 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.342842 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-dshm\") pod \"a17df043-eb89-468a-9fd0-c19e55353970\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " Jul 15 10:54:56.342958 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.342868 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jqg2\" (UniqueName: \"kubernetes.io/projected/a17df043-eb89-468a-9fd0-c19e55353970-kube-api-access-5jqg2\") pod \"a17df043-eb89-468a-9fd0-c19e55353970\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " Jul 15 10:54:56.342958 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.342910 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-model-cache\") pod \"a17df043-eb89-468a-9fd0-c19e55353970\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " Jul 15 10:54:56.342958 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.342936 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17df043-eb89-468a-9fd0-c19e55353970-tls-certs\") pod \"a17df043-eb89-468a-9fd0-c19e55353970\" (UID: \"a17df043-eb89-468a-9fd0-c19e55353970\") " Jul 15 10:54:56.343104 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.342962 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a17df043-eb89-468a-9fd0-c19e55353970" (UID: "a17df043-eb89-468a-9fd0-c19e55353970"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:54:56.343104 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.343045 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-home" (OuterVolumeSpecName: "home") pod "a17df043-eb89-468a-9fd0-c19e55353970" (UID: "a17df043-eb89-468a-9fd0-c19e55353970"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:54:56.343773 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.343685 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-tmp-dir\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:54:56.343891 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.343806 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-model-cache" (OuterVolumeSpecName: "model-cache") pod "a17df043-eb89-468a-9fd0-c19e55353970" (UID: "a17df043-eb89-468a-9fd0-c19e55353970"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:54:56.347787 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.343957 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-home\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:54:56.355191 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.355116 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-dshm" (OuterVolumeSpecName: "dshm") pod "a17df043-eb89-468a-9fd0-c19e55353970" (UID: "a17df043-eb89-468a-9fd0-c19e55353970"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:54:56.355327 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.355297 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a17df043-eb89-468a-9fd0-c19e55353970-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a17df043-eb89-468a-9fd0-c19e55353970" (UID: "a17df043-eb89-468a-9fd0-c19e55353970"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 10:54:56.357153 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.357127 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17df043-eb89-468a-9fd0-c19e55353970-kube-api-access-5jqg2" (OuterVolumeSpecName: "kube-api-access-5jqg2") pod "a17df043-eb89-468a-9fd0-c19e55353970" (UID: "a17df043-eb89-468a-9fd0-c19e55353970"). InnerVolumeSpecName "kube-api-access-5jqg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:54:56.444313 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.444290 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-dshm\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:54:56.444413 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.444329 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jqg2\" (UniqueName: \"kubernetes.io/projected/a17df043-eb89-468a-9fd0-c19e55353970-kube-api-access-5jqg2\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:54:56.444413 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.444338 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17df043-eb89-468a-9fd0-c19e55353970-model-cache\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:54:56.444413 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.444347 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17df043-eb89-468a-9fd0-c19e55353970-tls-certs\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:54:56.957939 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.957901 2579 generic.go:356] "Generic (PLEG): container finished" podID="a17df043-eb89-468a-9fd0-c19e55353970" containerID="0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6" exitCode=0 Jul 15 10:54:56.958105 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.957968 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" Jul 15 10:54:56.958105 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.957978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" event={"ID":"a17df043-eb89-468a-9fd0-c19e55353970","Type":"ContainerDied","Data":"0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6"} Jul 15 10:54:56.958105 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.958017 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689" event={"ID":"a17df043-eb89-468a-9fd0-c19e55353970","Type":"ContainerDied","Data":"dbf4ee8285e2915e0d17fd424041b09491f939e1882c6f1cad5856fddf01cea6"} Jul 15 10:54:56.958105 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.958040 2579 scope.go:117] "RemoveContainer" containerID="0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6" Jul 15 10:54:56.966440 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.966418 2579 scope.go:117] "RemoveContainer" containerID="0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6" Jul 15 10:54:56.966704 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:54:56.966687 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6\": container with ID starting with 0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6 not found: ID does not exist" containerID="0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6" Jul 15 10:54:56.966794 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.966712 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6"} err="failed to get container status \"0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6\": rpc error: code = NotFound desc = could not find container \"0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6\": container with ID starting with 0f663923f3d86cca19ca460137983f1d8270970330fd3719cc7c6c0f994217c6 not found: ID does not exist" Jul 15 10:54:56.979661 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.979626 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689"] Jul 15 10:54:56.986848 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:56.986820 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e595ab4b/e2e-lifecycle-model-e595ab4b-kserve-75559c4855-6z689"] Jul 15 10:54:58.351480 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:58.351447 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a17df043-eb89-468a-9fd0-c19e55353970" path="/var/lib/kubelet/pods/a17df043-eb89-468a-9fd0-c19e55353970/volumes" Jul 15 10:54:58.413376 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:58.413354 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:54:58.418964 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:54:58.418943 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:55:17.119450 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:17.119395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:18.941947 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:18.941900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:19.430942 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:19.430907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:25.779480 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:25.779424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:32.356017 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:32.355979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:32.458317 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:32.458283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:38.530150 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:38.530117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:38.741915 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:38.741878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:39.798756 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:39.798700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:39.900143 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:39.900116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:40.019545 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:40.019519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:40.715312 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:40.715281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:40.813995 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:40.813961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:40.915358 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:40.915329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:42.350806 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:42.350778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:48.952983 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:48.952895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:49.020025 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:49.019998 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:49.649301 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:49.649264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:56.461998 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:56.461958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:56.572051 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:56.572016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:56.662434 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:55:56.662401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:03.673627 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:03.673590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:03.774552 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:03.774516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:07.216509 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:07.216468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:07.570496 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:07.570395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:28.584134 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:28.584098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:35.230965 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:35.230927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:35.331477 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:35.331445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:35.614101 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:35.614061 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:35.951547 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:35.951454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:36.407372 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:36.407334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:38.670789 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:38.670729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:38.814210 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:38.814177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:38.917247 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:38.917210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:39.220611 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:39.220580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:39.443831 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:39.443798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:39.539616 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:39.539524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:43.176634 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:43.176590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:43.530166 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:43.530072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:43.882827 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:43.882791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:44.026371 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:44.026333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:46.745598 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:46.745553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:46.862709 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:46.862666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:46.955912 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:46.955871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:47.085658 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:47.085619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:47.916285 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:47.916250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:48.030904 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:48.030870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:51.170587 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:51.170542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:56.391286 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.391245 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh"] Jul 15 10:56:56.391671 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.391614 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17df043-eb89-468a-9fd0-c19e55353970" containerName="main" Jul 15 10:56:56.391671 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.391627 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17df043-eb89-468a-9fd0-c19e55353970" containerName="main" Jul 15 10:56:56.391778 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.391691 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17df043-eb89-468a-9fd0-c19e55353970" containerName="main" Jul 15 10:56:56.391778 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.391703 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26290275-1be4-4492-9752-eb83bb6acf26" containerName="istio-proxy" Jul 15 10:56:56.394469 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.394454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.398045 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.398012 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-97bab6\"/\"kube-root-ca.crt\"" Jul 15 10:56:56.398889 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.398867 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-97bab6\"/\"openshift-service-ca.crt\"" Jul 15 10:56:56.398984 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.398889 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-97bab6\"/\"default-dockercfg-9c6zz\"" Jul 15 10:56:56.399190 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.399173 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-97bab6\"/\"test-model-97bab6-kserve-self-signed-certs\"" Jul 15 10:56:56.405794 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.405768 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh"] Jul 15 10:56:56.424625 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.424602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:56.528966 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.528939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-home\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.528966 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.528968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-dshm\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.529142 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.528995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-tmp-dir\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.529142 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.529076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508922b9-c495-464f-9fea-684b9636fd1e-tls-certs\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.529142 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.529100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4xh\" (UniqueName: \"kubernetes.io/projected/508922b9-c495-464f-9fea-684b9636fd1e-kube-api-access-lb4xh\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.529142 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.529126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-model-cache\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.630008 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.629981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4xh\" (UniqueName: \"kubernetes.io/projected/508922b9-c495-464f-9fea-684b9636fd1e-kube-api-access-lb4xh\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.630129 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.630015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-model-cache\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.630129 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.630049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-home\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.630245 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.630178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-dshm\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.630245 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.630229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-tmp-dir\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.630356 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.630299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508922b9-c495-464f-9fea-684b9636fd1e-tls-certs\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.630412 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.630395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-model-cache\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.630467 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.630436 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-home\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.630592 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.630574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-tmp-dir\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.632360 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.632332 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-dshm\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.632705 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.632688 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508922b9-c495-464f-9fea-684b9636fd1e-tls-certs\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.640364 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.640342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4xh\" (UniqueName: \"kubernetes.io/projected/508922b9-c495-464f-9fea-684b9636fd1e-kube-api-access-lb4xh\") pod \"test-model-97bab6-kserve-58c5bdc459-h8fmh\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.704982 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.704919 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:56:56.841686 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:56.841615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh"] Jul 15 10:56:56.843970 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:56:56.843940 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod508922b9_c495_464f_9fea_684b9636fd1e.slice/crio-5d28b820cbea9f396a430ced99376fc01e9c92617ef61a2e7056a071ee00d895 WatchSource:0}: Error finding container 5d28b820cbea9f396a430ced99376fc01e9c92617ef61a2e7056a071ee00d895: Status 404 returned error can't find the container with id 5d28b820cbea9f396a430ced99376fc01e9c92617ef61a2e7056a071ee00d895 Jul 15 10:56:57.504930 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:57.504887 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" event={"ID":"508922b9-c495-464f-9fea-684b9636fd1e","Type":"ContainerStarted","Data":"62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297"} Jul 15 10:56:57.505289 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:57.504938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" event={"ID":"508922b9-c495-464f-9fea-684b9636fd1e","Type":"ContainerStarted","Data":"5d28b820cbea9f396a430ced99376fc01e9c92617ef61a2e7056a071ee00d895"} Jul 15 10:56:57.525078 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:56:57.525027 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" podStartSLOduration=1.525008724 podStartE2EDuration="1.525008724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:56:57.52318335 +0000 UTC m=+2219.656376689" watchObservedRunningTime="2026-07-15 10:56:57.525008724 +0000 UTC m=+2219.658202063" Jul 15 10:57:00.426798 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:00.426764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:00.529846 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:00.529816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:06.705968 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:06.705917 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:57:06.706459 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:06.706207 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:57:06.718841 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:06.718806 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:57:07.563868 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:07.563828 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:57:10.755916 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:10.755880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:10.854151 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:10.854119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:25.527261 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:25.527177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:45.027076 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:45.027036 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh"] Jul 15 10:57:45.051659 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:45.051631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:46.087671 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.087637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:46.229359 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.229322 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw"] Jul 15 10:57:46.232631 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.232609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw" Jul 15 10:57:46.235156 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.235132 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 10:57:46.235258 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.235132 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zj826\"" Jul 15 10:57:46.244404 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.244381 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw"] Jul 15 10:57:46.356317 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.356284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfde8a05-5d88-4804-be00-2d5f0e6399e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-97bab6-984dw\" (UID: \"cfde8a05-5d88-4804-be00-2d5f0e6399e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw" Jul 15 10:57:46.457492 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.457438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfde8a05-5d88-4804-be00-2d5f0e6399e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-97bab6-984dw\" (UID: \"cfde8a05-5d88-4804-be00-2d5f0e6399e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw" Jul 15 10:57:46.458119 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.458096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfde8a05-5d88-4804-be00-2d5f0e6399e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-97bab6-984dw\" (UID: \"cfde8a05-5d88-4804-be00-2d5f0e6399e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw" Jul 15 10:57:46.543064 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.543030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw" Jul 15 10:57:46.689935 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.689906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw"] Jul 15 10:57:46.691895 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:57:46.691864 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfde8a05_5d88_4804_be00_2d5f0e6399e5.slice/crio-4c848a28d47867c2f36eea2a93c02c4842f78121f41a4cd5cf660c0b811e9124 WatchSource:0}: Error finding container 4c848a28d47867c2f36eea2a93c02c4842f78121f41a4cd5cf660c0b811e9124: Status 404 returned error can't find the container with id 4c848a28d47867c2f36eea2a93c02c4842f78121f41a4cd5cf660c0b811e9124 Jul 15 10:57:46.728803 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:46.728769 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw" event={"ID":"cfde8a05-5d88-4804-be00-2d5f0e6399e5","Type":"ContainerStarted","Data":"4c848a28d47867c2f36eea2a93c02c4842f78121f41a4cd5cf660c0b811e9124"} Jul 15 10:57:47.106123 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:47.106088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:47.210607 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:47.210507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:47.734813 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:47.734771 2579 generic.go:356] "Generic (PLEG): container finished" podID="cfde8a05-5d88-4804-be00-2d5f0e6399e5" containerID="e1a649d307b24b4f7d3211ba1e205f1533d083557d2d7d7e80712d5d297fc19b" exitCode=0 Jul 15 10:57:47.735003 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:47.734822 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw" event={"ID":"cfde8a05-5d88-4804-be00-2d5f0e6399e5","Type":"ContainerDied","Data":"e1a649d307b24b4f7d3211ba1e205f1533d083557d2d7d7e80712d5d297fc19b"} Jul 15 10:57:48.880183 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:48.880159 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw" Jul 15 10:57:48.980054 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:48.980019 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfde8a05-5d88-4804-be00-2d5f0e6399e5-maas-api-service-ca\") pod \"cfde8a05-5d88-4804-be00-2d5f0e6399e5\" (UID: \"cfde8a05-5d88-4804-be00-2d5f0e6399e5\") " Jul 15 10:57:48.980398 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:48.980374 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfde8a05-5d88-4804-be00-2d5f0e6399e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfde8a05-5d88-4804-be00-2d5f0e6399e5" (UID: "cfde8a05-5d88-4804-be00-2d5f0e6399e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 10:57:49.081778 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:49.081666 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfde8a05-5d88-4804-be00-2d5f0e6399e5-maas-api-service-ca\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:57:49.745001 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:49.744967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw" event={"ID":"cfde8a05-5d88-4804-be00-2d5f0e6399e5","Type":"ContainerDied","Data":"4c848a28d47867c2f36eea2a93c02c4842f78121f41a4cd5cf660c0b811e9124"} Jul 15 10:57:49.745001 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:49.745001 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-97bab6-984dw" Jul 15 10:57:49.745211 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:49.745024 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c848a28d47867c2f36eea2a93c02c4842f78121f41a4cd5cf660c0b811e9124" Jul 15 10:57:55.685392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:55.685352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:56.824685 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:57:56.824646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:00.047431 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.047388 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" podUID="508922b9-c495-464f-9fea-684b9636fd1e" containerName="main" containerID="cri-o://62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297" gracePeriod=45 Jul 15 10:58:00.297494 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.297440 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:58:00.378438 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.378412 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-dshm\") pod \"508922b9-c495-464f-9fea-684b9636fd1e\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " Jul 15 10:58:00.378566 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.378468 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508922b9-c495-464f-9fea-684b9636fd1e-tls-certs\") pod \"508922b9-c495-464f-9fea-684b9636fd1e\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " Jul 15 10:58:00.378566 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.378506 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb4xh\" (UniqueName: \"kubernetes.io/projected/508922b9-c495-464f-9fea-684b9636fd1e-kube-api-access-lb4xh\") pod \"508922b9-c495-464f-9fea-684b9636fd1e\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " Jul 15 10:58:00.378566 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.378539 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-tmp-dir\") pod \"508922b9-c495-464f-9fea-684b9636fd1e\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " Jul 15 10:58:00.378566 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.378557 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-model-cache\") pod \"508922b9-c495-464f-9fea-684b9636fd1e\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " Jul 15 10:58:00.378808 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.378668 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-home\") pod \"508922b9-c495-464f-9fea-684b9636fd1e\" (UID: \"508922b9-c495-464f-9fea-684b9636fd1e\") " Jul 15 10:58:00.378876 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.378830 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "508922b9-c495-464f-9fea-684b9636fd1e" (UID: "508922b9-c495-464f-9fea-684b9636fd1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:58:00.378876 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.378852 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "508922b9-c495-464f-9fea-684b9636fd1e" (UID: "508922b9-c495-464f-9fea-684b9636fd1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:58:00.378993 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.378968 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-home" (OuterVolumeSpecName: "home") pod "508922b9-c495-464f-9fea-684b9636fd1e" (UID: "508922b9-c495-464f-9fea-684b9636fd1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:58:00.379175 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.379103 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-tmp-dir\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:58:00.379175 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.379131 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-model-cache\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:58:00.379175 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.379146 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-home\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:58:00.380907 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.380883 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508922b9-c495-464f-9fea-684b9636fd1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "508922b9-c495-464f-9fea-684b9636fd1e" (UID: "508922b9-c495-464f-9fea-684b9636fd1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 10:58:00.380995 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.380946 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508922b9-c495-464f-9fea-684b9636fd1e-kube-api-access-lb4xh" (OuterVolumeSpecName: "kube-api-access-lb4xh") pod "508922b9-c495-464f-9fea-684b9636fd1e" (UID: "508922b9-c495-464f-9fea-684b9636fd1e"). InnerVolumeSpecName "kube-api-access-lb4xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:58:00.380995 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.380964 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-dshm" (OuterVolumeSpecName: "dshm") pod "508922b9-c495-464f-9fea-684b9636fd1e" (UID: "508922b9-c495-464f-9fea-684b9636fd1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:58:00.479472 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.479447 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508922b9-c495-464f-9fea-684b9636fd1e-dshm\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:58:00.479472 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.479470 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508922b9-c495-464f-9fea-684b9636fd1e-tls-certs\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:58:00.479611 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.479480 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb4xh\" (UniqueName: \"kubernetes.io/projected/508922b9-c495-464f-9fea-684b9636fd1e-kube-api-access-lb4xh\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:58:00.796199 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.796170 2579 generic.go:356] "Generic (PLEG): container finished" podID="508922b9-c495-464f-9fea-684b9636fd1e" containerID="62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297" exitCode=0 Jul 15 10:58:00.796310 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.796211 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" Jul 15 10:58:00.796310 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.796236 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" event={"ID":"508922b9-c495-464f-9fea-684b9636fd1e","Type":"ContainerDied","Data":"62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297"} Jul 15 10:58:00.796310 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.796266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh" event={"ID":"508922b9-c495-464f-9fea-684b9636fd1e","Type":"ContainerDied","Data":"5d28b820cbea9f396a430ced99376fc01e9c92617ef61a2e7056a071ee00d895"} Jul 15 10:58:00.796310 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.796282 2579 scope.go:117] "RemoveContainer" containerID="62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297" Jul 15 10:58:00.805823 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.805805 2579 scope.go:117] "RemoveContainer" containerID="62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297" Jul 15 10:58:00.806057 ip-10-0-133-247 kubenswrapper[2579]: E0715 10:58:00.806040 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297\": container with ID starting with 62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297 not found: ID does not exist" containerID="62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297" Jul 15 10:58:00.806117 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.806070 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297"} err="failed to get container status \"62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297\": rpc error: code = NotFound desc = could not find container \"62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297\": container with ID starting with 62f8f2728b0f6064e2f2e240e55e0635049c5ec43ba78844dc5d3ed5acb2d297 not found: ID does not exist" Jul 15 10:58:00.822434 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.822406 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh"] Jul 15 10:58:00.826933 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:00.826914 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-97bab6/test-model-97bab6-kserve-58c5bdc459-h8fmh"] Jul 15 10:58:01.373017 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:01.372981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:02.350671 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:02.350640 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="508922b9-c495-464f-9fea-684b9636fd1e" path="/var/lib/kubelet/pods/508922b9-c495-464f-9fea-684b9636fd1e/volumes" Jul 15 10:58:09.053508 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:09.053476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:09.150898 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:09.150874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:12.173693 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.173649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:12.372939 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.372911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:12.504369 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.504291 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7"] Jul 15 10:58:12.504638 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.504626 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508922b9-c495-464f-9fea-684b9636fd1e" containerName="main" Jul 15 10:58:12.504685 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.504640 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="508922b9-c495-464f-9fea-684b9636fd1e" containerName="main" Jul 15 10:58:12.504685 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.504657 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfde8a05-5d88-4804-be00-2d5f0e6399e5" containerName="revoke-keys" Jul 15 10:58:12.504685 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.504663 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfde8a05-5d88-4804-be00-2d5f0e6399e5" containerName="revoke-keys" Jul 15 10:58:12.504809 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.504717 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfde8a05-5d88-4804-be00-2d5f0e6399e5" containerName="revoke-keys" Jul 15 10:58:12.504809 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.504727 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="508922b9-c495-464f-9fea-684b9636fd1e" containerName="main" Jul 15 10:58:12.507615 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.507596 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7" Jul 15 10:58:12.510934 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.510914 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 10:58:12.511034 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.510989 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zj826\"" Jul 15 10:58:12.520041 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.520019 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7"] Jul 15 10:58:12.571820 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.571796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a313961-08ba-4d3a-8ad4-6d25c32bcfeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7\" (UID: \"8a313961-08ba-4d3a-8ad4-6d25c32bcfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7" Jul 15 10:58:12.672539 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.672515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a313961-08ba-4d3a-8ad4-6d25c32bcfeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7\" (UID: \"8a313961-08ba-4d3a-8ad4-6d25c32bcfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7" Jul 15 10:58:12.673122 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.673099 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a313961-08ba-4d3a-8ad4-6d25c32bcfeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7\" (UID: \"8a313961-08ba-4d3a-8ad4-6d25c32bcfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7" Jul 15 10:58:12.817623 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.817539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7" Jul 15 10:58:12.950798 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:12.950770 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7"] Jul 15 10:58:12.953345 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:58:12.953323 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a313961_08ba_4d3a_8ad4_6d25c32bcfeb.slice/crio-6a4c6f2f3027c469f4f49acdef7310f1b27ebeb02b1fbc3b9629badc2f2cc287 WatchSource:0}: Error finding container 6a4c6f2f3027c469f4f49acdef7310f1b27ebeb02b1fbc3b9629badc2f2cc287: Status 404 returned error can't find the container with id 6a4c6f2f3027c469f4f49acdef7310f1b27ebeb02b1fbc3b9629badc2f2cc287 Jul 15 10:58:13.271580 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:13.271554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:13.375894 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:13.375868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:13.863784 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:13.863729 2579 generic.go:356] "Generic (PLEG): container finished" podID="8a313961-08ba-4d3a-8ad4-6d25c32bcfeb" containerID="6e25bc48a759b7d28d27a4a951af373d245e60a425e3318a28c4cadd3d7b3cd8" exitCode=0 Jul 15 10:58:13.863921 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:13.863821 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7" event={"ID":"8a313961-08ba-4d3a-8ad4-6d25c32bcfeb","Type":"ContainerDied","Data":"6e25bc48a759b7d28d27a4a951af373d245e60a425e3318a28c4cadd3d7b3cd8"} Jul 15 10:58:13.863921 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:13.863889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7" event={"ID":"8a313961-08ba-4d3a-8ad4-6d25c32bcfeb","Type":"ContainerStarted","Data":"6a4c6f2f3027c469f4f49acdef7310f1b27ebeb02b1fbc3b9629badc2f2cc287"} Jul 15 10:58:14.996108 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:14.996078 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7" Jul 15 10:58:15.093030 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:15.093007 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a313961-08ba-4d3a-8ad4-6d25c32bcfeb-maas-api-service-ca\") pod \"8a313961-08ba-4d3a-8ad4-6d25c32bcfeb\" (UID: \"8a313961-08ba-4d3a-8ad4-6d25c32bcfeb\") " Jul 15 10:58:15.093362 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:15.093340 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a313961-08ba-4d3a-8ad4-6d25c32bcfeb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a313961-08ba-4d3a-8ad4-6d25c32bcfeb" (UID: "8a313961-08ba-4d3a-8ad4-6d25c32bcfeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 10:58:15.193987 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:15.193931 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a313961-08ba-4d3a-8ad4-6d25c32bcfeb-maas-api-service-ca\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 10:58:15.874782 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:15.874729 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7" Jul 15 10:58:15.875028 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:15.874769 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7" event={"ID":"8a313961-08ba-4d3a-8ad4-6d25c32bcfeb","Type":"ContainerDied","Data":"6a4c6f2f3027c469f4f49acdef7310f1b27ebeb02b1fbc3b9629badc2f2cc287"} Jul 15 10:58:15.875028 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:15.874835 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a4c6f2f3027c469f4f49acdef7310f1b27ebeb02b1fbc3b9629badc2f2cc287" Jul 15 10:58:23.307060 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:23.307028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:28.025704 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:28.025663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:35.005759 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:35.005709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:35.106591 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:35.106562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:38.447560 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:38.447526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:38.655896 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:38.655859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:39.022355 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:39.022321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:39.403049 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:39.403023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:39.884068 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:39.884034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:42.329484 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:42.329452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:42.436362 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:42.436330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:42.542407 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:42.542372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:42.642261 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:42.642231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:43.156394 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:43.156366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:43.259129 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:43.259104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:44.264129 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:44.264101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:44.335941 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:44.335912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:45.521053 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:45.521018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:45.615228 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:45.615201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:46.739369 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:46.739339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:47.542732 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.542705 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z"] Jul 15 10:58:47.543133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.543116 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a313961-08ba-4d3a-8ad4-6d25c32bcfeb" containerName="revoke-keys" Jul 15 10:58:47.543133 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.543136 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a313961-08ba-4d3a-8ad4-6d25c32bcfeb" containerName="revoke-keys" Jul 15 10:58:47.543312 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.543211 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a313961-08ba-4d3a-8ad4-6d25c32bcfeb" containerName="revoke-keys" Jul 15 10:58:47.546392 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.546372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.549707 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.549690 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-25c606-openshift-default-dockercfg-s9nsn\"" Jul 15 10:58:47.564365 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.564340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z"] Jul 15 10:58:47.647185 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.647156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-socket\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.647313 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.647203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da567194-2648-4fa9-a08f-278095fc7f65-istio-podinfo\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.647313 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.647240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-certs\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.647313 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.647284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmk8m\" (UniqueName: \"kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-kube-api-access-dmk8m\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.647501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.647344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-envoy\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.647501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.647368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-data\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.647501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.647399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da567194-2648-4fa9-a08f-278095fc7f65-istiod-ca-cert\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.647501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.647464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-credential-socket\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.647501 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.647492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-istio-token\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.748393 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-certs\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.748821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmk8m\" (UniqueName: \"kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-kube-api-access-dmk8m\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.748821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-envoy\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.748821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-data\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.748821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da567194-2648-4fa9-a08f-278095fc7f65-istiod-ca-cert\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.748821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-credential-socket\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.748821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-istio-token\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.748821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-socket\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.748821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da567194-2648-4fa9-a08f-278095fc7f65-istio-podinfo\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.749243 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-certs\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.749243 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.748913 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-data\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.749243 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.749137 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-socket\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.749243 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.749166 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-credential-socket\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.749385 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.749303 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da567194-2648-4fa9-a08f-278095fc7f65-istiod-ca-cert\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.750827 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.750806 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-envoy\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.751411 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.751395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da567194-2648-4fa9-a08f-278095fc7f65-istio-podinfo\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.758764 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.758719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-istio-token\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.759149 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.759125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmk8m\" (UniqueName: \"kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-kube-api-access-dmk8m\") pod \"e2e-api-b-25c606-openshift-default-5659c6777c-mht5z\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:47.857970 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:47.857944 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:48.001255 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:48.001217 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z"] Jul 15 10:58:48.004966 ip-10-0-133-247 kubenswrapper[2579]: W0715 10:58:48.004930 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda567194_2648_4fa9_a08f_278095fc7f65.slice/crio-ed39a8a1b9ac3f398439f851ea13c38ab642cb50c152db88afdf86c77fec3e6b WatchSource:0}: Error finding container ed39a8a1b9ac3f398439f851ea13c38ab642cb50c152db88afdf86c77fec3e6b: Status 404 returned error can't find the container with id ed39a8a1b9ac3f398439f851ea13c38ab642cb50c152db88afdf86c77fec3e6b Jul 15 10:58:48.007103 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:48.007072 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:58:48.007205 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:48.007173 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:58:48.007264 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:48.007224 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:58:48.028731 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:48.028707 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" event={"ID":"da567194-2648-4fa9-a08f-278095fc7f65","Type":"ContainerStarted","Data":"ed39a8a1b9ac3f398439f851ea13c38ab642cb50c152db88afdf86c77fec3e6b"} Jul 15 10:58:48.159364 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:48.159280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:49.034646 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:49.034611 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" event={"ID":"da567194-2648-4fa9-a08f-278095fc7f65","Type":"ContainerStarted","Data":"aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab"} Jul 15 10:58:49.061954 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:49.061908 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" podStartSLOduration=2.061894949 podStartE2EDuration="2.061894949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:58:49.059967284 +0000 UTC m=+2331.193160622" watchObservedRunningTime="2026-07-15 10:58:49.061894949 +0000 UTC m=+2331.195088287" Jul 15 10:58:49.859071 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:49.859035 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:49.864578 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:49.864547 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:50.040174 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:50.040147 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:50.040867 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:50.040848 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:58:51.026656 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:51.026626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:51.130357 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:51.130324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:51.203465 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:51.203436 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:51.456400 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:51.456363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:51.938041 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:51.938011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:52.055759 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:52.055693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:55.558895 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:58:55.558862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:04.114297 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:04.114231 2579 generic.go:356] "Generic (PLEG): container finished" podID="da567194-2648-4fa9-a08f-278095fc7f65" containerID="aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab" exitCode=0 Jul 15 10:59:04.114638 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:04.114307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" event={"ID":"da567194-2648-4fa9-a08f-278095fc7f65","Type":"ContainerDied","Data":"aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab"} Jul 15 10:59:04.114683 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:04.114663 2579 scope.go:117] "RemoveContainer" containerID="aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab" Jul 15 10:59:04.115691 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:04.115661 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:59:04.115821 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:04.115769 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:59:04.115890 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:04.115821 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 10:59:04.434813 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:04.434719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:04.534185 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:04.534157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:05.040591 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:05.040556 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:59:05.120717 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:05.120683 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" event={"ID":"da567194-2648-4fa9-a08f-278095fc7f65","Type":"ContainerStarted","Data":"9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2"} Jul 15 10:59:05.859054 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:05.859024 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:59:05.859797 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:05.859778 2579 patch_prober.go:28] interesting pod/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 10:59:05.859850 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:05.859825 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 10:59:06.858902 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:06.858866 2579 patch_prober.go:28] interesting pod/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 10:59:06.859266 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:06.858929 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 10:59:07.858728 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:07.858688 2579 patch_prober.go:28] interesting pod/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 10:59:07.858930 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:07.858771 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 10:59:08.858844 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:08.858805 2579 patch_prober.go:28] interesting pod/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 10:59:08.859040 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:08.858870 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 10:59:09.862539 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:09.862508 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:59:09.862956 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:09.862781 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:59:09.863507 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:09.863488 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 10:59:12.993124 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:12.993093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:13.090567 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:13.090534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:13.918874 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:13.918841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:17.898052 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:17.898020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:18.788102 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:18.788071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:25.028875 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:25.028837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:25.128655 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:25.128623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:29.517090 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:29.517061 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:29.745557 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:29.745524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:40.560801 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:40.560759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:44.998255 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:44.998220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:52.294125 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:52.294083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:52.395475 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:52.395424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:55.709172 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:55.709137 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z"] Jul 15 10:59:55.709628 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:55.709394 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" containerID="cri-o://9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2" gracePeriod=30 Jul 15 10:59:55.736809 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:55.736770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:55.979912 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:55.979830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:56.342507 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:56.342420 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:56.680580 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:56.680552 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:57.403012 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:57.402979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:58.444814 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:58.444778 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:59:58.450350 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:58.450330 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 10:59:59.681262 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:59.681227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:59.757383 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:59.757352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 10:59:59.991145 ip-10-0-133-247 kubenswrapper[2579]: I0715 10:59:59.991072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:00.220332 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:00.220294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:00.912694 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:00.912658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:00.961526 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:00.961504 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 11:00:01.034301 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.034218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:01.126558 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.126530 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da567194-2648-4fa9-a08f-278095fc7f65-istio-podinfo\") pod \"da567194-2648-4fa9-a08f-278095fc7f65\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " Jul 15 11:00:01.126726 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.126576 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da567194-2648-4fa9-a08f-278095fc7f65-istiod-ca-cert\") pod \"da567194-2648-4fa9-a08f-278095fc7f65\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " Jul 15 11:00:01.126726 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.126608 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-credential-socket\") pod \"da567194-2648-4fa9-a08f-278095fc7f65\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " Jul 15 11:00:01.126902 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.126771 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-socket\") pod \"da567194-2648-4fa9-a08f-278095fc7f65\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " Jul 15 11:00:01.126902 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.126817 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-envoy\") pod \"da567194-2648-4fa9-a08f-278095fc7f65\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " Jul 15 11:00:01.126902 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.126841 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-data\") pod \"da567194-2648-4fa9-a08f-278095fc7f65\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " Jul 15 11:00:01.126902 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.126872 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-certs\") pod \"da567194-2648-4fa9-a08f-278095fc7f65\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " Jul 15 11:00:01.127110 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.126912 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-istio-token\") pod \"da567194-2648-4fa9-a08f-278095fc7f65\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " Jul 15 11:00:01.127110 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.126915 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da567194-2648-4fa9-a08f-278095fc7f65" (UID: "da567194-2648-4fa9-a08f-278095fc7f65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:00:01.127110 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.126973 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmk8m\" (UniqueName: \"kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-kube-api-access-dmk8m\") pod \"da567194-2648-4fa9-a08f-278095fc7f65\" (UID: \"da567194-2648-4fa9-a08f-278095fc7f65\") " Jul 15 11:00:01.127110 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.127063 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da567194-2648-4fa9-a08f-278095fc7f65" (UID: "da567194-2648-4fa9-a08f-278095fc7f65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:00:01.127110 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.127094 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da567194-2648-4fa9-a08f-278095fc7f65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da567194-2648-4fa9-a08f-278095fc7f65" (UID: "da567194-2648-4fa9-a08f-278095fc7f65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:00:01.127386 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.127211 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da567194-2648-4fa9-a08f-278095fc7f65" (UID: "da567194-2648-4fa9-a08f-278095fc7f65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:00:01.127386 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.127321 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-credential-socket\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:00:01.127386 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.127340 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-socket\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:00:01.127386 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.127355 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-workload-certs\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:00:01.127386 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.127370 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da567194-2648-4fa9-a08f-278095fc7f65-istiod-ca-cert\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:00:01.127969 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.127939 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-data" (OuterVolumeSpecName: "istio-data") pod "da567194-2648-4fa9-a08f-278095fc7f65" (UID: "da567194-2648-4fa9-a08f-278095fc7f65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:00:01.129342 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.129316 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-istio-token" (OuterVolumeSpecName: "istio-token") pod "da567194-2648-4fa9-a08f-278095fc7f65" (UID: "da567194-2648-4fa9-a08f-278095fc7f65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:00:01.129462 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.129354 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da567194-2648-4fa9-a08f-278095fc7f65" (UID: "da567194-2648-4fa9-a08f-278095fc7f65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:00:01.129530 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.129496 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da567194-2648-4fa9-a08f-278095fc7f65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da567194-2648-4fa9-a08f-278095fc7f65" (UID: "da567194-2648-4fa9-a08f-278095fc7f65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 11:00:01.129724 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.129705 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-kube-api-access-dmk8m" (OuterVolumeSpecName: "kube-api-access-dmk8m") pod "da567194-2648-4fa9-a08f-278095fc7f65" (UID: "da567194-2648-4fa9-a08f-278095fc7f65"). InnerVolumeSpecName "kube-api-access-dmk8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:00:01.227947 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.227913 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-envoy\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:00:01.227947 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.227939 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da567194-2648-4fa9-a08f-278095fc7f65-istio-data\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:00:01.227947 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.227947 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-istio-token\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:00:01.228193 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.227958 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmk8m\" (UniqueName: \"kubernetes.io/projected/da567194-2648-4fa9-a08f-278095fc7f65-kube-api-access-dmk8m\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:00:01.228193 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.227968 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da567194-2648-4fa9-a08f-278095fc7f65-istio-podinfo\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:00:01.394499 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.394467 2579 generic.go:356] "Generic (PLEG): container finished" podID="da567194-2648-4fa9-a08f-278095fc7f65" containerID="9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2" exitCode=0 Jul 15 11:00:01.394661 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.394515 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" event={"ID":"da567194-2648-4fa9-a08f-278095fc7f65","Type":"ContainerDied","Data":"9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2"} Jul 15 11:00:01.394661 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.394521 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" Jul 15 11:00:01.394661 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.394560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z" event={"ID":"da567194-2648-4fa9-a08f-278095fc7f65","Type":"ContainerDied","Data":"ed39a8a1b9ac3f398439f851ea13c38ab642cb50c152db88afdf86c77fec3e6b"} Jul 15 11:00:01.394661 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.394582 2579 scope.go:117] "RemoveContainer" containerID="9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2" Jul 15 11:00:01.404026 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.404009 2579 scope.go:117] "RemoveContainer" containerID="aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab" Jul 15 11:00:01.412041 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.412019 2579 scope.go:117] "RemoveContainer" containerID="9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2" Jul 15 11:00:01.412295 ip-10-0-133-247 kubenswrapper[2579]: E0715 11:00:01.412274 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2\": container with ID starting with 9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2 not found: ID does not exist" containerID="9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2" Jul 15 11:00:01.412352 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.412303 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2"} err="failed to get container status \"9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2\": rpc error: code = NotFound desc = could not find container \"9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2\": container with ID starting with 9b180a1af38bc1321f3527f5e0d30f3c9ef489d11dbeaf12dc2e99ce42f9a6b2 not found: ID does not exist" Jul 15 11:00:01.412352 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.412322 2579 scope.go:117] "RemoveContainer" containerID="aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab" Jul 15 11:00:01.412551 ip-10-0-133-247 kubenswrapper[2579]: E0715 11:00:01.412533 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab\": container with ID starting with aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab not found: ID does not exist" containerID="aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab" Jul 15 11:00:01.412598 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.412558 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab"} err="failed to get container status \"aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab\": rpc error: code = NotFound desc = could not find container \"aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab\": container with ID starting with aec7e676dc5de23b3e3bc303925b42a857d85f982bbc4858209be7106d04dbab not found: ID does not exist" Jul 15 11:00:01.428125 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.428098 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z"] Jul 15 11:00:01.443725 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:01.443699 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-25c606-openshift-default-5659c6777c-mht5z"] Jul 15 11:00:02.351382 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:02.351343 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da567194-2648-4fa9-a08f-278095fc7f65" path="/var/lib/kubelet/pods/da567194-2648-4fa9-a08f-278095fc7f65/volumes" Jul 15 11:00:04.222640 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:04.222605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:04.562923 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:04.562843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:04.950491 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:04.950453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:05.145096 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:05.145065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:07.945681 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:07.945644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:08.044521 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:08.044488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:08.217668 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:08.217577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:08.484398 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:08.484315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:09.339615 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:09.339584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:09.442475 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:09.442444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:12.358846 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:12.358815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:21.402157 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.402071 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr"] Jul 15 11:00:21.402602 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.402583 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" Jul 15 11:00:21.402674 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.402606 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" Jul 15 11:00:21.402674 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.402630 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" Jul 15 11:00:21.402674 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.402640 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" Jul 15 11:00:21.402855 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.402725 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" Jul 15 11:00:21.405626 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.405605 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.408877 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.408847 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b2a040\"/\"kube-root-ca.crt\"" Jul 15 11:00:21.408958 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.408875 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b2a040\"/\"openshift-service-ca.crt\"" Jul 15 11:00:21.408958 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.408847 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b2a040\"/\"auth-test-model-b2a040-kserve-self-signed-certs\"" Jul 15 11:00:21.409037 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.408887 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b2a040\"/\"default-dockercfg-c8298\"" Jul 15 11:00:21.427861 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.427839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:21.429175 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.429157 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr"] Jul 15 11:00:21.484365 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.484344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-home\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.484473 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.484380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-dshm\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.484473 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.484406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-tmp-dir\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.484473 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.484465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x98xf\" (UniqueName: \"kubernetes.io/projected/dcc4baa7-5116-4646-b068-f2ffaae20b88-kube-api-access-x98xf\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.484606 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.484554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-model-cache\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.484606 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.484576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc4baa7-5116-4646-b068-f2ffaae20b88-tls-certs\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.585169 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.585139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-home\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.585296 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.585173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-dshm\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.585296 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.585191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-tmp-dir\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.585389 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.585337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98xf\" (UniqueName: \"kubernetes.io/projected/dcc4baa7-5116-4646-b068-f2ffaae20b88-kube-api-access-x98xf\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.585439 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.585422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-model-cache\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.585492 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.585451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc4baa7-5116-4646-b068-f2ffaae20b88-tls-certs\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.585643 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.585623 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-home\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.585714 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.585676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-tmp-dir\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.585815 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.585800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-model-cache\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.587492 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.587471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-dshm\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.587887 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.587871 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc4baa7-5116-4646-b068-f2ffaae20b88-tls-certs\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.594764 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.594714 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x98xf\" (UniqueName: \"kubernetes.io/projected/dcc4baa7-5116-4646-b068-f2ffaae20b88-kube-api-access-x98xf\") pod \"auth-test-model-b2a040-kserve-79444fc5c9-wv8lr\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.714299 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.714232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:21.851179 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:21.851147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr"] Jul 15 11:00:21.854306 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:00:21.854280 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc4baa7_5116_4646_b068_f2ffaae20b88.slice/crio-5dadff2a92ecdc91a4167f311d9cbce3316baf6b5cd0a46c23b95ed87211484c WatchSource:0}: Error finding container 5dadff2a92ecdc91a4167f311d9cbce3316baf6b5cd0a46c23b95ed87211484c: Status 404 returned error can't find the container with id 5dadff2a92ecdc91a4167f311d9cbce3316baf6b5cd0a46c23b95ed87211484c Jul 15 11:00:22.198522 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:22.198486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:22.298854 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:22.298824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:22.498922 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:22.498837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" event={"ID":"dcc4baa7-5116-4646-b068-f2ffaae20b88","Type":"ContainerStarted","Data":"50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e"} Jul 15 11:00:22.498922 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:22.498880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" event={"ID":"dcc4baa7-5116-4646-b068-f2ffaae20b88","Type":"ContainerStarted","Data":"5dadff2a92ecdc91a4167f311d9cbce3316baf6b5cd0a46c23b95ed87211484c"} Jul 15 11:00:22.518267 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:22.518220 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" podStartSLOduration=1.518206867 podStartE2EDuration="1.518206867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:00:22.517324618 +0000 UTC m=+2424.650517955" watchObservedRunningTime="2026-07-15 11:00:22.518206867 +0000 UTC m=+2424.651400205" Jul 15 11:00:30.477611 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:30.477576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:30.578193 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:30.578162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:00:31.714733 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:31.714699 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:31.715196 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:31.714771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:31.727281 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:31.727261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:00:32.553152 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:00:32.553122 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:01:03.032252 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:01:03.032216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:01:11.188900 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:01:11.188864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:01:11.281129 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:01:11.281096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:01:12.300808 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:01:12.300765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:01:53.231147 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:01:53.231053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:03:03.686498 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:03.686463 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr"] Jul 15 11:03:03.708811 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:03.708770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:03:04.419839 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:04.419805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:03:04.536988 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:04.536937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:03:11.175254 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.175222 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m"] Jul 15 11:03:11.175719 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.175603 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da567194-2648-4fa9-a08f-278095fc7f65" containerName="istio-proxy" Jul 15 11:03:11.178337 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.178318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.181183 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.181160 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b2a040\"/\"sub-test-model-b2a040-kserve-self-signed-certs\"" Jul 15 11:03:11.188445 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.188423 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m"] Jul 15 11:03:11.206191 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.206165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:03:11.226363 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.226332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tmp-dir\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.226501 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.226380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-model-cache\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.226501 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.226413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-dshm\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.226501 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.226465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tls-certs\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.226627 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.226506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vptcl\" (UniqueName: \"kubernetes.io/projected/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-kube-api-access-vptcl\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.226627 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.226543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-home\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.327825 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.327798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tmp-dir\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.327958 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.327837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-model-cache\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.327958 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.327861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-dshm\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.327958 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.327877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tls-certs\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.327958 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.327896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptcl\" (UniqueName: \"kubernetes.io/projected/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-kube-api-access-vptcl\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.328138 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.328041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-home\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.328297 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.328270 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tmp-dir\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.328558 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.328519 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-model-cache\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.328828 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.328801 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-home\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.330627 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.330606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tls-certs\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.330724 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.330645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-dshm\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.338651 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.338629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vptcl\" (UniqueName: \"kubernetes.io/projected/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-kube-api-access-vptcl\") pod \"sub-test-model-b2a040-kserve-7c554695fc-7pm9m\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.487866 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.487801 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:11.617657 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:11.617632 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m"] Jul 15 11:03:11.619384 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:03:11.619358 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5aac3f5f_2384_4e0c_97a6_6ebbec85016c.slice/crio-8d9b1af83c0ecfb424c285e44177f1a853b839b8df5ac02035b3c0bcda85a6bf WatchSource:0}: Error finding container 8d9b1af83c0ecfb424c285e44177f1a853b839b8df5ac02035b3c0bcda85a6bf: Status 404 returned error can't find the container with id 8d9b1af83c0ecfb424c285e44177f1a853b839b8df5ac02035b3c0bcda85a6bf Jul 15 11:03:12.287430 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:12.287396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" event={"ID":"5aac3f5f-2384-4e0c-97a6-6ebbec85016c","Type":"ContainerStarted","Data":"20036d81366be614dc6585ccb57aa50b7c3ad1457e8918ab8edb32040fc02b00"} Jul 15 11:03:12.287430 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:12.287438 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" event={"ID":"5aac3f5f-2384-4e0c-97a6-6ebbec85016c","Type":"ContainerStarted","Data":"8d9b1af83c0ecfb424c285e44177f1a853b839b8df5ac02035b3c0bcda85a6bf"} Jul 15 11:03:12.320916 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:12.320872 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" podStartSLOduration=1.32085788 podStartE2EDuration="1.32085788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:03:12.319418632 +0000 UTC m=+2594.452611970" watchObservedRunningTime="2026-07-15 11:03:12.32085788 +0000 UTC m=+2594.454051217" Jul 15 11:03:18.708636 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.708548 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" podUID="dcc4baa7-5116-4646-b068-f2ffaae20b88" containerName="main" containerID="cri-o://50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e" gracePeriod=45 Jul 15 11:03:18.965250 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.965180 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:03:18.995226 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.995199 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-dshm\") pod \"dcc4baa7-5116-4646-b068-f2ffaae20b88\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " Jul 15 11:03:18.995343 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.995259 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-home\") pod \"dcc4baa7-5116-4646-b068-f2ffaae20b88\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " Jul 15 11:03:18.995343 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.995290 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc4baa7-5116-4646-b068-f2ffaae20b88-tls-certs\") pod \"dcc4baa7-5116-4646-b068-f2ffaae20b88\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " Jul 15 11:03:18.995343 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.995318 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x98xf\" (UniqueName: \"kubernetes.io/projected/dcc4baa7-5116-4646-b068-f2ffaae20b88-kube-api-access-x98xf\") pod \"dcc4baa7-5116-4646-b068-f2ffaae20b88\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " Jul 15 11:03:18.995511 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.995355 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-tmp-dir\") pod \"dcc4baa7-5116-4646-b068-f2ffaae20b88\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " Jul 15 11:03:18.995511 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.995391 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-model-cache\") pod \"dcc4baa7-5116-4646-b068-f2ffaae20b88\" (UID: \"dcc4baa7-5116-4646-b068-f2ffaae20b88\") " Jul 15 11:03:18.995634 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.995596 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-home" (OuterVolumeSpecName: "home") pod "dcc4baa7-5116-4646-b068-f2ffaae20b88" (UID: "dcc4baa7-5116-4646-b068-f2ffaae20b88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:03:18.995775 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.995721 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dcc4baa7-5116-4646-b068-f2ffaae20b88" (UID: "dcc4baa7-5116-4646-b068-f2ffaae20b88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:03:18.995775 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.995765 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-home\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:03:18.995921 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.995844 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-model-cache" (OuterVolumeSpecName: "model-cache") pod "dcc4baa7-5116-4646-b068-f2ffaae20b88" (UID: "dcc4baa7-5116-4646-b068-f2ffaae20b88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:03:18.997570 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.997546 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-dshm" (OuterVolumeSpecName: "dshm") pod "dcc4baa7-5116-4646-b068-f2ffaae20b88" (UID: "dcc4baa7-5116-4646-b068-f2ffaae20b88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:03:18.997672 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.997573 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc4baa7-5116-4646-b068-f2ffaae20b88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dcc4baa7-5116-4646-b068-f2ffaae20b88" (UID: "dcc4baa7-5116-4646-b068-f2ffaae20b88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:03:18.997723 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:18.997665 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc4baa7-5116-4646-b068-f2ffaae20b88-kube-api-access-x98xf" (OuterVolumeSpecName: "kube-api-access-x98xf") pod "dcc4baa7-5116-4646-b068-f2ffaae20b88" (UID: "dcc4baa7-5116-4646-b068-f2ffaae20b88"). InnerVolumeSpecName "kube-api-access-x98xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:03:19.096332 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.096310 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-dshm\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:03:19.096332 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.096333 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc4baa7-5116-4646-b068-f2ffaae20b88-tls-certs\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:03:19.096458 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.096343 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x98xf\" (UniqueName: \"kubernetes.io/projected/dcc4baa7-5116-4646-b068-f2ffaae20b88-kube-api-access-x98xf\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:03:19.096458 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.096352 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-tmp-dir\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:03:19.096458 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.096363 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc4baa7-5116-4646-b068-f2ffaae20b88-model-cache\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:03:19.322595 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.322535 2579 generic.go:356] "Generic (PLEG): container finished" podID="dcc4baa7-5116-4646-b068-f2ffaae20b88" containerID="50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e" exitCode=0 Jul 15 11:03:19.322595 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.322579 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" Jul 15 11:03:19.322595 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.322582 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" event={"ID":"dcc4baa7-5116-4646-b068-f2ffaae20b88","Type":"ContainerDied","Data":"50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e"} Jul 15 11:03:19.322809 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.322614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr" event={"ID":"dcc4baa7-5116-4646-b068-f2ffaae20b88","Type":"ContainerDied","Data":"5dadff2a92ecdc91a4167f311d9cbce3316baf6b5cd0a46c23b95ed87211484c"} Jul 15 11:03:19.322809 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.322631 2579 scope.go:117] "RemoveContainer" containerID="50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e" Jul 15 11:03:19.331488 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.331472 2579 scope.go:117] "RemoveContainer" containerID="50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e" Jul 15 11:03:19.331718 ip-10-0-133-247 kubenswrapper[2579]: E0715 11:03:19.331701 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e\": container with ID starting with 50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e not found: ID does not exist" containerID="50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e" Jul 15 11:03:19.331823 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.331724 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e"} err="failed to get container status \"50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e\": rpc error: code = NotFound desc = could not find container \"50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e\": container with ID starting with 50d32f54818f9d72f9e49f82fbdb08e191c169ce849986de04aa6f9296328b0e not found: ID does not exist" Jul 15 11:03:19.344993 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.344967 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr"] Jul 15 11:03:19.350484 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:19.350457 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/auth-test-model-b2a040-kserve-79444fc5c9-wv8lr"] Jul 15 11:03:20.351051 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:20.351012 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcc4baa7-5116-4646-b068-f2ffaae20b88" path="/var/lib/kubelet/pods/dcc4baa7-5116-4646-b068-f2ffaae20b88/volumes" Jul 15 11:03:21.488297 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:21.488261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:21.488665 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:21.488313 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:21.501232 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:21.501202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:22.350943 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:22.350917 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:03:34.171938 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.171896 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs"] Jul 15 11:03:34.172450 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.172430 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcc4baa7-5116-4646-b068-f2ffaae20b88" containerName="main" Jul 15 11:03:34.172533 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.172452 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcc4baa7-5116-4646-b068-f2ffaae20b88" containerName="main" Jul 15 11:03:34.172589 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.172532 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcc4baa7-5116-4646-b068-f2ffaae20b88" containerName="main" Jul 15 11:03:34.177059 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.177039 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.180962 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.180939 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ceb9f0\"/\"openshift-service-ca.crt\"" Jul 15 11:03:34.181074 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.180972 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ceb9f0\"/\"default-dockercfg-99mbq\"" Jul 15 11:03:34.181074 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.180972 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ceb9f0\"/\"kube-root-ca.crt\"" Jul 15 11:03:34.181074 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.180993 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ceb9f0\"/\"sub-test-model-ceb9f0-kserve-self-signed-certs\"" Jul 15 11:03:34.187553 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.187501 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs"] Jul 15 11:03:34.201154 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.201135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:03:34.317170 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.317144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7bb716f-ee82-414a-8072-f3890f798194-tls-certs\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.317315 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.317189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-tmp-dir\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.317315 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.317208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-model-cache\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.317315 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.317230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwsd\" (UniqueName: \"kubernetes.io/projected/b7bb716f-ee82-414a-8072-f3890f798194-kube-api-access-dpwsd\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.317315 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.317264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-dshm\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.317315 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.317280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-home\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.417709 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.417677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-home\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.417847 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.417800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7bb716f-ee82-414a-8072-f3890f798194-tls-certs\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.417924 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.417849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-tmp-dir\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.417924 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.417873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-model-cache\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.418032 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.417915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwsd\" (UniqueName: \"kubernetes.io/projected/b7bb716f-ee82-414a-8072-f3890f798194-kube-api-access-dpwsd\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.418032 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.417952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-dshm\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.418136 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.418054 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-home\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.418196 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.418148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-model-cache\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.418253 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.418212 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-tmp-dir\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.420128 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.420100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-dshm\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.420472 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.420453 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7bb716f-ee82-414a-8072-f3890f798194-tls-certs\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.429082 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.429026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwsd\" (UniqueName: \"kubernetes.io/projected/b7bb716f-ee82-414a-8072-f3890f798194-kube-api-access-dpwsd\") pod \"sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.488320 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.488297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:34.618346 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:34.618318 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs"] Jul 15 11:03:34.620190 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:03:34.620166 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7bb716f_ee82_414a_8072_f3890f798194.slice/crio-6bdfd3ed6c703bede4d4c6d8848dcf87fed613aefca52be1b03a9c53e5056257 WatchSource:0}: Error finding container 6bdfd3ed6c703bede4d4c6d8848dcf87fed613aefca52be1b03a9c53e5056257: Status 404 returned error can't find the container with id 6bdfd3ed6c703bede4d4c6d8848dcf87fed613aefca52be1b03a9c53e5056257 Jul 15 11:03:35.399065 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:35.399033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" event={"ID":"b7bb716f-ee82-414a-8072-f3890f798194","Type":"ContainerStarted","Data":"9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201"} Jul 15 11:03:35.399590 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:35.399074 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" event={"ID":"b7bb716f-ee82-414a-8072-f3890f798194","Type":"ContainerStarted","Data":"6bdfd3ed6c703bede4d4c6d8848dcf87fed613aefca52be1b03a9c53e5056257"} Jul 15 11:03:35.419249 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:35.419205 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" podStartSLOduration=1.419190693 podStartE2EDuration="1.419190693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:03:35.417061949 +0000 UTC m=+2617.550255288" watchObservedRunningTime="2026-07-15 11:03:35.419190693 +0000 UTC m=+2617.552384070" Jul 15 11:03:44.489470 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:44.489437 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:44.489871 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:44.489482 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:44.502014 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:44.501990 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:03:45.284942 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:45.284905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:03:45.382164 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:45.382137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:03:45.458464 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:03:45.458435 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:04:16.432438 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:16.432407 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m"] Jul 15 11:04:16.443520 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:16.443491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:04:17.330594 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:17.330561 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs"] Jul 15 11:04:17.672171 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:17.672139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:04:26.615788 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:26.615734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:04:31.453314 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.453256 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" podUID="5aac3f5f-2384-4e0c-97a6-6ebbec85016c" containerName="main" containerID="cri-o://20036d81366be614dc6585ccb57aa50b7c3ad1457e8918ab8edb32040fc02b00" gracePeriod=45 Jul 15 11:04:31.666219 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.666188 2579 generic.go:356] "Generic (PLEG): container finished" podID="5aac3f5f-2384-4e0c-97a6-6ebbec85016c" containerID="20036d81366be614dc6585ccb57aa50b7c3ad1457e8918ab8edb32040fc02b00" exitCode=0 Jul 15 11:04:31.666357 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.666271 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" event={"ID":"5aac3f5f-2384-4e0c-97a6-6ebbec85016c","Type":"ContainerDied","Data":"20036d81366be614dc6585ccb57aa50b7c3ad1457e8918ab8edb32040fc02b00"} Jul 15 11:04:31.704132 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.704075 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:04:31.784223 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784199 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-dshm\") pod \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " Jul 15 11:04:31.784370 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784233 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-model-cache\") pod \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " Jul 15 11:04:31.784370 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784257 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tls-certs\") pod \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " Jul 15 11:04:31.784370 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784306 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tmp-dir\") pod \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " Jul 15 11:04:31.784370 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784365 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-home\") pod \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " Jul 15 11:04:31.784586 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784475 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5aac3f5f-2384-4e0c-97a6-6ebbec85016c" (UID: "5aac3f5f-2384-4e0c-97a6-6ebbec85016c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:04:31.784586 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784513 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vptcl\" (UniqueName: \"kubernetes.io/projected/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-kube-api-access-vptcl\") pod \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\" (UID: \"5aac3f5f-2384-4e0c-97a6-6ebbec85016c\") " Jul 15 11:04:31.784685 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784610 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-home" (OuterVolumeSpecName: "home") pod "5aac3f5f-2384-4e0c-97a6-6ebbec85016c" (UID: "5aac3f5f-2384-4e0c-97a6-6ebbec85016c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:04:31.784685 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784625 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5aac3f5f-2384-4e0c-97a6-6ebbec85016c" (UID: "5aac3f5f-2384-4e0c-97a6-6ebbec85016c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:04:31.784821 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784806 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-home\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:31.784890 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784822 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-model-cache\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:31.784890 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.784833 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tmp-dir\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:31.786528 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.786499 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-dshm" (OuterVolumeSpecName: "dshm") pod "5aac3f5f-2384-4e0c-97a6-6ebbec85016c" (UID: "5aac3f5f-2384-4e0c-97a6-6ebbec85016c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:04:31.786664 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.786554 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5aac3f5f-2384-4e0c-97a6-6ebbec85016c" (UID: "5aac3f5f-2384-4e0c-97a6-6ebbec85016c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:04:31.786664 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.786658 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-kube-api-access-vptcl" (OuterVolumeSpecName: "kube-api-access-vptcl") pod "5aac3f5f-2384-4e0c-97a6-6ebbec85016c" (UID: "5aac3f5f-2384-4e0c-97a6-6ebbec85016c"). InnerVolumeSpecName "kube-api-access-vptcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:04:31.885586 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.885565 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-dshm\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:31.885586 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.885585 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-tls-certs\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:31.885710 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:31.885596 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vptcl\" (UniqueName: \"kubernetes.io/projected/5aac3f5f-2384-4e0c-97a6-6ebbec85016c-kube-api-access-vptcl\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:32.350675 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.350633 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" podUID="b7bb716f-ee82-414a-8072-f3890f798194" containerName="main" containerID="cri-o://9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201" gracePeriod=45 Jul 15 11:04:32.597470 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.597444 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:04:32.671628 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.671563 2579 generic.go:356] "Generic (PLEG): container finished" podID="b7bb716f-ee82-414a-8072-f3890f798194" containerID="9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201" exitCode=0 Jul 15 11:04:32.671628 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.671606 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" Jul 15 11:04:32.671861 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.671649 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" event={"ID":"b7bb716f-ee82-414a-8072-f3890f798194","Type":"ContainerDied","Data":"9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201"} Jul 15 11:04:32.671861 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.671705 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs" event={"ID":"b7bb716f-ee82-414a-8072-f3890f798194","Type":"ContainerDied","Data":"6bdfd3ed6c703bede4d4c6d8848dcf87fed613aefca52be1b03a9c53e5056257"} Jul 15 11:04:32.671861 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.671727 2579 scope.go:117] "RemoveContainer" containerID="9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201" Jul 15 11:04:32.672972 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.672952 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" event={"ID":"5aac3f5f-2384-4e0c-97a6-6ebbec85016c","Type":"ContainerDied","Data":"8d9b1af83c0ecfb424c285e44177f1a853b839b8df5ac02035b3c0bcda85a6bf"} Jul 15 11:04:32.673052 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.672985 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m" Jul 15 11:04:32.680245 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.680223 2579 scope.go:117] "RemoveContainer" containerID="9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201" Jul 15 11:04:32.680511 ip-10-0-133-247 kubenswrapper[2579]: E0715 11:04:32.680492 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201\": container with ID starting with 9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201 not found: ID does not exist" containerID="9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201" Jul 15 11:04:32.680587 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.680523 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201"} err="failed to get container status \"9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201\": rpc error: code = NotFound desc = could not find container \"9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201\": container with ID starting with 9874f996b7681bbbb146663d11c6792febd9095dc7e6dd3f3b592dc09fe34201 not found: ID does not exist" Jul 15 11:04:32.680587 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.680549 2579 scope.go:117] "RemoveContainer" containerID="20036d81366be614dc6585ccb57aa50b7c3ad1457e8918ab8edb32040fc02b00" Jul 15 11:04:32.691483 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.691461 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m"] Jul 15 11:04:32.692971 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.692955 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-model-cache\") pod \"b7bb716f-ee82-414a-8072-f3890f798194\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " Jul 15 11:04:32.693041 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.692992 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-tmp-dir\") pod \"b7bb716f-ee82-414a-8072-f3890f798194\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " Jul 15 11:04:32.693041 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.693029 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7bb716f-ee82-414a-8072-f3890f798194-tls-certs\") pod \"b7bb716f-ee82-414a-8072-f3890f798194\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " Jul 15 11:04:32.693153 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.693066 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-home\") pod \"b7bb716f-ee82-414a-8072-f3890f798194\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " Jul 15 11:04:32.693153 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.693094 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-dshm\") pod \"b7bb716f-ee82-414a-8072-f3890f798194\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " Jul 15 11:04:32.693153 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.693131 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpwsd\" (UniqueName: \"kubernetes.io/projected/b7bb716f-ee82-414a-8072-f3890f798194-kube-api-access-dpwsd\") pod \"b7bb716f-ee82-414a-8072-f3890f798194\" (UID: \"b7bb716f-ee82-414a-8072-f3890f798194\") " Jul 15 11:04:32.693305 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.693267 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7bb716f-ee82-414a-8072-f3890f798194" (UID: "b7bb716f-ee82-414a-8072-f3890f798194"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:04:32.693305 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.693285 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7bb716f-ee82-414a-8072-f3890f798194" (UID: "b7bb716f-ee82-414a-8072-f3890f798194"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:04:32.693393 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.693311 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-home" (OuterVolumeSpecName: "home") pod "b7bb716f-ee82-414a-8072-f3890f798194" (UID: "b7bb716f-ee82-414a-8072-f3890f798194"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:04:32.693511 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.693486 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-home\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:32.693613 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.693512 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-model-cache\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:32.693613 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.693527 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-tmp-dir\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:32.695725 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.695689 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bb716f-ee82-414a-8072-f3890f798194-kube-api-access-dpwsd" (OuterVolumeSpecName: "kube-api-access-dpwsd") pod "b7bb716f-ee82-414a-8072-f3890f798194" (UID: "b7bb716f-ee82-414a-8072-f3890f798194"). InnerVolumeSpecName "kube-api-access-dpwsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:04:32.695989 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.695965 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7bb716f-ee82-414a-8072-f3890f798194-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7bb716f-ee82-414a-8072-f3890f798194" (UID: "b7bb716f-ee82-414a-8072-f3890f798194"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:04:32.696085 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.696022 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-dshm" (OuterVolumeSpecName: "dshm") pod "b7bb716f-ee82-414a-8072-f3890f798194" (UID: "b7bb716f-ee82-414a-8072-f3890f798194"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:04:32.697238 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.697221 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b2a040/sub-test-model-b2a040-kserve-7c554695fc-7pm9m"] Jul 15 11:04:32.794556 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.794529 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpwsd\" (UniqueName: \"kubernetes.io/projected/b7bb716f-ee82-414a-8072-f3890f798194-kube-api-access-dpwsd\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:32.794556 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.794553 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7bb716f-ee82-414a-8072-f3890f798194-tls-certs\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:32.794673 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.794566 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7bb716f-ee82-414a-8072-f3890f798194-dshm\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:04:32.994984 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.994961 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs"] Jul 15 11:04:32.999544 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:32.999525 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/sub-test-model-ceb9f0-kserve-564fcb6f65-tm6qs"] Jul 15 11:04:34.351589 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:34.351552 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aac3f5f-2384-4e0c-97a6-6ebbec85016c" path="/var/lib/kubelet/pods/5aac3f5f-2384-4e0c-97a6-6ebbec85016c/volumes" Jul 15 11:04:34.351997 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:34.351962 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7bb716f-ee82-414a-8072-f3890f798194" path="/var/lib/kubelet/pods/b7bb716f-ee82-414a-8072-f3890f798194/volumes" Jul 15 11:04:36.900875 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:36.900846 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:04:36.968936 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:36.968910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:04:45.783474 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.783441 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68"] Jul 15 11:04:45.784147 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.783945 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aac3f5f-2384-4e0c-97a6-6ebbec85016c" containerName="main" Jul 15 11:04:45.784147 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.783965 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aac3f5f-2384-4e0c-97a6-6ebbec85016c" containerName="main" Jul 15 11:04:45.784147 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.783978 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7bb716f-ee82-414a-8072-f3890f798194" containerName="main" Jul 15 11:04:45.784147 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.783986 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7bb716f-ee82-414a-8072-f3890f798194" containerName="main" Jul 15 11:04:45.784147 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.784119 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aac3f5f-2384-4e0c-97a6-6ebbec85016c" containerName="main" Jul 15 11:04:45.784147 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.784131 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7bb716f-ee82-414a-8072-f3890f798194" containerName="main" Jul 15 11:04:45.786936 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.786913 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.790153 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.790130 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ceb9f0\"/\"default-dockercfg-99mbq\"" Jul 15 11:04:45.790272 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.790178 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ceb9f0\"/\"kube-root-ca.crt\"" Jul 15 11:04:45.790272 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.790182 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ceb9f0\"/\"rate-test-model-ceb9f0-kserve-self-signed-certs\"" Jul 15 11:04:45.790377 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.790130 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ceb9f0\"/\"openshift-service-ca.crt\"" Jul 15 11:04:45.798312 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.798290 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68"] Jul 15 11:04:45.826427 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.826400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:04:45.887885 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.887863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b987aa86-e276-47b9-be66-4e370d9a0ea9-tls-certs\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.887980 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.887890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-tmp-dir\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.887980 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.887923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-home\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.888065 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.888004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-model-cache\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.888065 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.888035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-dshm\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.888169 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.888067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9fcc\" (UniqueName: \"kubernetes.io/projected/b987aa86-e276-47b9-be66-4e370d9a0ea9-kube-api-access-f9fcc\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.989158 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.989119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-model-cache\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.989278 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.989171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-dshm\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.989278 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.989210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9fcc\" (UniqueName: \"kubernetes.io/projected/b987aa86-e276-47b9-be66-4e370d9a0ea9-kube-api-access-f9fcc\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.989278 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.989269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b987aa86-e276-47b9-be66-4e370d9a0ea9-tls-certs\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.989444 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.989383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-tmp-dir\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.989444 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.989431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-home\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.989576 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.989555 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-model-cache\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.989714 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.989696 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-tmp-dir\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.989803 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.989769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-home\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.991609 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.991586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-dshm\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.991876 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.991839 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b987aa86-e276-47b9-be66-4e370d9a0ea9-tls-certs\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:45.999048 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:45.999026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9fcc\" (UniqueName: \"kubernetes.io/projected/b987aa86-e276-47b9-be66-4e370d9a0ea9-kube-api-access-f9fcc\") pod \"rate-test-model-ceb9f0-kserve-6d959fb755-wnl68\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:46.096546 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:46.096520 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:46.230205 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:46.230147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68"] Jul 15 11:04:46.232571 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:04:46.232547 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb987aa86_e276_47b9_be66_4e370d9a0ea9.slice/crio-f23930fb6b2215fd57f994d1c00dd761e983581972e0ab71196cc5481823414a WatchSource:0}: Error finding container f23930fb6b2215fd57f994d1c00dd761e983581972e0ab71196cc5481823414a: Status 404 returned error can't find the container with id f23930fb6b2215fd57f994d1c00dd761e983581972e0ab71196cc5481823414a Jul 15 11:04:46.742070 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:46.742037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" event={"ID":"b987aa86-e276-47b9-be66-4e370d9a0ea9","Type":"ContainerStarted","Data":"30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674"} Jul 15 11:04:46.742241 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:46.742082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" event={"ID":"b987aa86-e276-47b9-be66-4e370d9a0ea9","Type":"ContainerStarted","Data":"f23930fb6b2215fd57f994d1c00dd761e983581972e0ab71196cc5481823414a"} Jul 15 11:04:46.762914 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:46.762852 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" podStartSLOduration=1.762832764 podStartE2EDuration="1.762832764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:04:46.762597081 +0000 UTC m=+2688.895790421" watchObservedRunningTime="2026-07-15 11:04:46.762832764 +0000 UTC m=+2688.896026109" Jul 15 11:04:56.096759 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:56.096668 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:56.096759 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:56.096722 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:56.109461 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:56.109439 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:56.806649 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:56.806618 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:04:58.472735 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:58.472709 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 11:04:58.479095 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:04:58.479074 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-klwfw_492b2069-1e55-477f-b333-19f21a28a3f0/ovn-acl-logging/0.log" Jul 15 11:05:23.371877 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:23.371845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:23.472219 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:23.472186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:31.498929 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:31.498898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:31.600572 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:31.600539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:31.756981 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:31.756899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:32.684996 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:32.684960 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68"] Jul 15 11:05:32.705658 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:32.705625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:33.664133 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:33.664099 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-139754926732208"] Jul 15 11:05:33.667568 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:33.667551 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926732208" Jul 15 11:05:33.670141 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:33.670120 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dpnxh\"" Jul 15 11:05:33.679370 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:33.679349 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754926732208"] Jul 15 11:05:33.771231 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:33.771208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfrh7\" (UniqueName: \"kubernetes.io/projected/ed1fe76c-6701-4e11-8539-5000d175539d-kube-api-access-mfrh7\") pod \"test-curl-7520-139754926732208\" (UID: \"ed1fe76c-6701-4e11-8539-5000d175539d\") " pod="opendatahub/test-curl-7520-139754926732208" Jul 15 11:05:33.871562 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:33.871540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfrh7\" (UniqueName: \"kubernetes.io/projected/ed1fe76c-6701-4e11-8539-5000d175539d-kube-api-access-mfrh7\") pod \"test-curl-7520-139754926732208\" (UID: \"ed1fe76c-6701-4e11-8539-5000d175539d\") " pod="opendatahub/test-curl-7520-139754926732208" Jul 15 11:05:33.883421 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:33.883394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfrh7\" (UniqueName: \"kubernetes.io/projected/ed1fe76c-6701-4e11-8539-5000d175539d-kube-api-access-mfrh7\") pod \"test-curl-7520-139754926732208\" (UID: \"ed1fe76c-6701-4e11-8539-5000d175539d\") " pod="opendatahub/test-curl-7520-139754926732208" Jul 15 11:05:33.976501 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:33.976449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926732208" Jul 15 11:05:34.106778 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:34.106736 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754926732208"] Jul 15 11:05:34.108974 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:05:34.108943 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded1fe76c_6701_4e11_8539_5000d175539d.slice/crio-d2c06afc96ec6f340ce0351a55d6a8835cae752129ee6199e5a392f7f8e7d950 WatchSource:0}: Error finding container d2c06afc96ec6f340ce0351a55d6a8835cae752129ee6199e5a392f7f8e7d950: Status 404 returned error can't find the container with id d2c06afc96ec6f340ce0351a55d6a8835cae752129ee6199e5a392f7f8e7d950 Jul 15 11:05:34.110286 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:34.110266 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 11:05:34.967664 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:34.967632 2579 generic.go:356] "Generic (PLEG): container finished" podID="ed1fe76c-6701-4e11-8539-5000d175539d" containerID="3232982323d17a69beed81c4cc314b7dade40a73ea0929e82cbcae8adbc9dbba" exitCode=0 Jul 15 11:05:34.968005 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:34.967721 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754926732208" event={"ID":"ed1fe76c-6701-4e11-8539-5000d175539d","Type":"ContainerDied","Data":"3232982323d17a69beed81c4cc314b7dade40a73ea0929e82cbcae8adbc9dbba"} Jul 15 11:05:34.968005 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:34.967791 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754926732208" event={"ID":"ed1fe76c-6701-4e11-8539-5000d175539d","Type":"ContainerStarted","Data":"d2c06afc96ec6f340ce0351a55d6a8835cae752129ee6199e5a392f7f8e7d950"} Jul 15 11:05:36.099471 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.099448 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926732208" Jul 15 11:05:36.125839 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.125814 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7520-139754926732208_ed1fe76c-6701-4e11-8539-5000d175539d/test-curl-7520-139754926732208/0.log" Jul 15 11:05:36.154067 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.154042 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-139754926732208"] Jul 15 11:05:36.158792 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.158770 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-139754926732208"] Jul 15 11:05:36.191328 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.191297 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfrh7\" (UniqueName: \"kubernetes.io/projected/ed1fe76c-6701-4e11-8539-5000d175539d-kube-api-access-mfrh7\") pod \"ed1fe76c-6701-4e11-8539-5000d175539d\" (UID: \"ed1fe76c-6701-4e11-8539-5000d175539d\") " Jul 15 11:05:36.193505 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.193477 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1fe76c-6701-4e11-8539-5000d175539d-kube-api-access-mfrh7" (OuterVolumeSpecName: "kube-api-access-mfrh7") pod "ed1fe76c-6701-4e11-8539-5000d175539d" (UID: "ed1fe76c-6701-4e11-8539-5000d175539d"). InnerVolumeSpecName "kube-api-access-mfrh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:05:36.282822 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.282768 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-139754926733616"] Jul 15 11:05:36.283115 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.283100 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed1fe76c-6701-4e11-8539-5000d175539d" containerName="test-curl-7520-139754926732208" Jul 15 11:05:36.283171 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.283116 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed1fe76c-6701-4e11-8539-5000d175539d" containerName="test-curl-7520-139754926732208" Jul 15 11:05:36.283210 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.283185 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed1fe76c-6701-4e11-8539-5000d175539d" containerName="test-curl-7520-139754926732208" Jul 15 11:05:36.285918 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.285903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926733616" Jul 15 11:05:36.292117 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.291902 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfrh7\" (UniqueName: \"kubernetes.io/projected/ed1fe76c-6701-4e11-8539-5000d175539d-kube-api-access-mfrh7\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:36.295092 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.295064 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754926733616"] Jul 15 11:05:36.351677 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.351652 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed1fe76c-6701-4e11-8539-5000d175539d" path="/var/lib/kubelet/pods/ed1fe76c-6701-4e11-8539-5000d175539d/volumes" Jul 15 11:05:36.392717 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.392693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqdxf\" (UniqueName: \"kubernetes.io/projected/76dec743-1b8b-4f97-8f74-1b280578f116-kube-api-access-zqdxf\") pod \"test-curl-7520-139754926733616\" (UID: \"76dec743-1b8b-4f97-8f74-1b280578f116\") " pod="opendatahub/test-curl-7520-139754926733616" Jul 15 11:05:36.493359 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.493334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqdxf\" (UniqueName: \"kubernetes.io/projected/76dec743-1b8b-4f97-8f74-1b280578f116-kube-api-access-zqdxf\") pod \"test-curl-7520-139754926733616\" (UID: \"76dec743-1b8b-4f97-8f74-1b280578f116\") " pod="opendatahub/test-curl-7520-139754926733616" Jul 15 11:05:36.503955 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.503932 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqdxf\" (UniqueName: \"kubernetes.io/projected/76dec743-1b8b-4f97-8f74-1b280578f116-kube-api-access-zqdxf\") pod \"test-curl-7520-139754926733616\" (UID: \"76dec743-1b8b-4f97-8f74-1b280578f116\") " pod="opendatahub/test-curl-7520-139754926733616" Jul 15 11:05:36.595661 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.595643 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926733616" Jul 15 11:05:36.737271 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.737249 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754926733616"] Jul 15 11:05:36.739455 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:05:36.739428 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76dec743_1b8b_4f97_8f74_1b280578f116.slice/crio-3a3e01f158e0100a506236eaa0c2c0545c9fd6b6c0e9d61d5c162f2a3d88849b WatchSource:0}: Error finding container 3a3e01f158e0100a506236eaa0c2c0545c9fd6b6c0e9d61d5c162f2a3d88849b: Status 404 returned error can't find the container with id 3a3e01f158e0100a506236eaa0c2c0545c9fd6b6c0e9d61d5c162f2a3d88849b Jul 15 11:05:36.979098 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.979005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754926733616" event={"ID":"76dec743-1b8b-4f97-8f74-1b280578f116","Type":"ContainerStarted","Data":"3a3e01f158e0100a506236eaa0c2c0545c9fd6b6c0e9d61d5c162f2a3d88849b"} Jul 15 11:05:36.980223 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.980205 2579 scope.go:117] "RemoveContainer" containerID="3232982323d17a69beed81c4cc314b7dade40a73ea0929e82cbcae8adbc9dbba" Jul 15 11:05:36.980223 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:36.980216 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926732208" Jul 15 11:05:37.986221 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:37.986184 2579 generic.go:356] "Generic (PLEG): container finished" podID="76dec743-1b8b-4f97-8f74-1b280578f116" containerID="aa21c70a9f9b814548c55245224d9656431066f619e4d473a5ff07a8510d1bb2" exitCode=0 Jul 15 11:05:37.986633 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:37.986273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754926733616" event={"ID":"76dec743-1b8b-4f97-8f74-1b280578f116","Type":"ContainerDied","Data":"aa21c70a9f9b814548c55245224d9656431066f619e4d473a5ff07a8510d1bb2"} Jul 15 11:05:39.124774 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.124728 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926733616" Jul 15 11:05:39.158787 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.158733 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7520-139754926733616_76dec743-1b8b-4f97-8f74-1b280578f116/test-curl-7520-139754926733616/0.log" Jul 15 11:05:39.196695 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.196663 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-139754926733616"] Jul 15 11:05:39.201913 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.201891 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-139754926733616"] Jul 15 11:05:39.213298 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.213272 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqdxf\" (UniqueName: \"kubernetes.io/projected/76dec743-1b8b-4f97-8f74-1b280578f116-kube-api-access-zqdxf\") pod \"76dec743-1b8b-4f97-8f74-1b280578f116\" (UID: \"76dec743-1b8b-4f97-8f74-1b280578f116\") " Jul 15 11:05:39.215371 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.215349 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dec743-1b8b-4f97-8f74-1b280578f116-kube-api-access-zqdxf" (OuterVolumeSpecName: "kube-api-access-zqdxf") pod "76dec743-1b8b-4f97-8f74-1b280578f116" (UID: "76dec743-1b8b-4f97-8f74-1b280578f116"). InnerVolumeSpecName "kube-api-access-zqdxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:05:39.313959 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.313894 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqdxf\" (UniqueName: \"kubernetes.io/projected/76dec743-1b8b-4f97-8f74-1b280578f116-kube-api-access-zqdxf\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:39.367381 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.367358 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-139754926687984"] Jul 15 11:05:39.367690 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.367677 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76dec743-1b8b-4f97-8f74-1b280578f116" containerName="test-curl-7520-139754926733616" Jul 15 11:05:39.367753 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.367691 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="76dec743-1b8b-4f97-8f74-1b280578f116" containerName="test-curl-7520-139754926733616" Jul 15 11:05:39.367793 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.367759 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="76dec743-1b8b-4f97-8f74-1b280578f116" containerName="test-curl-7520-139754926733616" Jul 15 11:05:39.370546 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.370531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:39.379008 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.378987 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754926687984"] Jul 15 11:05:39.414838 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.414804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r726p\" (UniqueName: \"kubernetes.io/projected/6198949d-cc25-4cab-baef-627ddf281a0c-kube-api-access-r726p\") pod \"test-curl-7520-139754926687984\" (UID: \"6198949d-cc25-4cab-baef-627ddf281a0c\") " pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:39.515618 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.515592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r726p\" (UniqueName: \"kubernetes.io/projected/6198949d-cc25-4cab-baef-627ddf281a0c-kube-api-access-r726p\") pod \"test-curl-7520-139754926687984\" (UID: \"6198949d-cc25-4cab-baef-627ddf281a0c\") " pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:39.525925 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.525895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r726p\" (UniqueName: \"kubernetes.io/projected/6198949d-cc25-4cab-baef-627ddf281a0c-kube-api-access-r726p\") pod \"test-curl-7520-139754926687984\" (UID: \"6198949d-cc25-4cab-baef-627ddf281a0c\") " pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:39.679422 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.679401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:39.813342 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.813309 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754926687984"] Jul 15 11:05:39.816728 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:05:39.816693 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6198949d_cc25_4cab_baef_627ddf281a0c.slice/crio-8a206cf04872ad4c6186f540ac37d73045aa5b2f3ac670ac329d018053990514 WatchSource:0}: Error finding container 8a206cf04872ad4c6186f540ac37d73045aa5b2f3ac670ac329d018053990514: Status 404 returned error can't find the container with id 8a206cf04872ad4c6186f540ac37d73045aa5b2f3ac670ac329d018053990514 Jul 15 11:05:39.997550 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.997494 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a3e01f158e0100a506236eaa0c2c0545c9fd6b6c0e9d61d5c162f2a3d88849b" Jul 15 11:05:39.997550 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.997505 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926733616" Jul 15 11:05:39.998642 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:39.998619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754926687984" event={"ID":"6198949d-cc25-4cab-baef-627ddf281a0c","Type":"ContainerStarted","Data":"8a206cf04872ad4c6186f540ac37d73045aa5b2f3ac670ac329d018053990514"} Jul 15 11:05:40.008125 ip-10-0-133-247 kubenswrapper[2579]: E0715 11:05:40.008099 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7520-139754926733616\" is forbidden: User \"system:node:ip-10-0-133-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-247.ec2.internal' and this object" podUID="76dec743-1b8b-4f97-8f74-1b280578f116" pod="opendatahub/test-curl-7520-139754926733616" Jul 15 11:05:40.351841 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:40.351805 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76dec743-1b8b-4f97-8f74-1b280578f116" path="/var/lib/kubelet/pods/76dec743-1b8b-4f97-8f74-1b280578f116/volumes" Jul 15 11:05:41.005278 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:41.005239 2579 generic.go:356] "Generic (PLEG): container finished" podID="6198949d-cc25-4cab-baef-627ddf281a0c" containerID="a106a7127a89319760c61b7d63db90d23b8a760064ec9bac5c678ff40ece99dd" exitCode=0 Jul 15 11:05:41.005413 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:41.005287 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754926687984" event={"ID":"6198949d-cc25-4cab-baef-627ddf281a0c","Type":"ContainerDied","Data":"a106a7127a89319760c61b7d63db90d23b8a760064ec9bac5c678ff40ece99dd"} Jul 15 11:05:42.135408 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.135386 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:42.167160 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.167137 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7520-139754926687984_6198949d-cc25-4cab-baef-627ddf281a0c/test-curl-7520-139754926687984/0.log" Jul 15 11:05:42.205839 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.205812 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-139754926687984"] Jul 15 11:05:42.215476 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.215452 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-139754926687984"] Jul 15 11:05:42.236838 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.236816 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r726p\" (UniqueName: \"kubernetes.io/projected/6198949d-cc25-4cab-baef-627ddf281a0c-kube-api-access-r726p\") pod \"6198949d-cc25-4cab-baef-627ddf281a0c\" (UID: \"6198949d-cc25-4cab-baef-627ddf281a0c\") " Jul 15 11:05:42.238928 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.238908 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6198949d-cc25-4cab-baef-627ddf281a0c-kube-api-access-r726p" (OuterVolumeSpecName: "kube-api-access-r726p") pod "6198949d-cc25-4cab-baef-627ddf281a0c" (UID: "6198949d-cc25-4cab-baef-627ddf281a0c"). InnerVolumeSpecName "kube-api-access-r726p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:05:42.334345 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.334289 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-139754926687984"] Jul 15 11:05:42.334629 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.334616 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6198949d-cc25-4cab-baef-627ddf281a0c" containerName="test-curl-7520-139754926687984" Jul 15 11:05:42.334670 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.334630 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6198949d-cc25-4cab-baef-627ddf281a0c" containerName="test-curl-7520-139754926687984" Jul 15 11:05:42.334705 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.334694 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6198949d-cc25-4cab-baef-627ddf281a0c" containerName="test-curl-7520-139754926687984" Jul 15 11:05:42.337539 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.337524 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:42.337917 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.337902 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r726p\" (UniqueName: \"kubernetes.io/projected/6198949d-cc25-4cab-baef-627ddf281a0c-kube-api-access-r726p\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:42.344045 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.344023 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754926687984"] Jul 15 11:05:42.350767 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.350721 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6198949d-cc25-4cab-baef-627ddf281a0c" path="/var/lib/kubelet/pods/6198949d-cc25-4cab-baef-627ddf281a0c/volumes" Jul 15 11:05:42.438748 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.438725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfnl9\" (UniqueName: \"kubernetes.io/projected/d3591ed5-dcce-46ac-b225-18116eeef308-kube-api-access-nfnl9\") pod \"test-curl-7520-139754926687984\" (UID: \"d3591ed5-dcce-46ac-b225-18116eeef308\") " pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:42.539280 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.539256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfnl9\" (UniqueName: \"kubernetes.io/projected/d3591ed5-dcce-46ac-b225-18116eeef308-kube-api-access-nfnl9\") pod \"test-curl-7520-139754926687984\" (UID: \"d3591ed5-dcce-46ac-b225-18116eeef308\") " pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:42.548653 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.548626 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfnl9\" (UniqueName: \"kubernetes.io/projected/d3591ed5-dcce-46ac-b225-18116eeef308-kube-api-access-nfnl9\") pod \"test-curl-7520-139754926687984\" (UID: \"d3591ed5-dcce-46ac-b225-18116eeef308\") " pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:42.647793 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.647770 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:42.783387 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:42.783359 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754926687984"] Jul 15 11:05:42.786941 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:05:42.786911 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3591ed5_dcce_46ac_b225_18116eeef308.slice/crio-835875ad285bec42c9d51f584a2c6243ae5f533315a2e88beeaf7ead8a81dacc WatchSource:0}: Error finding container 835875ad285bec42c9d51f584a2c6243ae5f533315a2e88beeaf7ead8a81dacc: Status 404 returned error can't find the container with id 835875ad285bec42c9d51f584a2c6243ae5f533315a2e88beeaf7ead8a81dacc Jul 15 11:05:43.015950 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:43.015876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754926687984" event={"ID":"d3591ed5-dcce-46ac-b225-18116eeef308","Type":"ContainerStarted","Data":"835875ad285bec42c9d51f584a2c6243ae5f533315a2e88beeaf7ead8a81dacc"} Jul 15 11:05:43.017182 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:43.017164 2579 scope.go:117] "RemoveContainer" containerID="a106a7127a89319760c61b7d63db90d23b8a760064ec9bac5c678ff40ece99dd" Jul 15 11:05:43.017288 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:43.017170 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:44.024228 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:44.024188 2579 generic.go:356] "Generic (PLEG): container finished" podID="d3591ed5-dcce-46ac-b225-18116eeef308" containerID="727b9838abe916e7a5912f6bd9f613b149a3b828ac524e540efb020ad33d7223" exitCode=0 Jul 15 11:05:44.024684 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:44.024259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754926687984" event={"ID":"d3591ed5-dcce-46ac-b225-18116eeef308","Type":"ContainerDied","Data":"727b9838abe916e7a5912f6bd9f613b149a3b828ac524e540efb020ad33d7223"} Jul 15 11:05:45.182932 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.182905 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:45.211434 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.211409 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7520-139754926687984_d3591ed5-dcce-46ac-b225-18116eeef308/test-curl-7520-139754926687984/0.log" Jul 15 11:05:45.242075 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.242025 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-139754926687984"] Jul 15 11:05:45.245196 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.245176 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-139754926687984"] Jul 15 11:05:45.259383 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.259358 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfnl9\" (UniqueName: \"kubernetes.io/projected/d3591ed5-dcce-46ac-b225-18116eeef308-kube-api-access-nfnl9\") pod \"d3591ed5-dcce-46ac-b225-18116eeef308\" (UID: \"d3591ed5-dcce-46ac-b225-18116eeef308\") " Jul 15 11:05:45.261479 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.261456 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3591ed5-dcce-46ac-b225-18116eeef308-kube-api-access-nfnl9" (OuterVolumeSpecName: "kube-api-access-nfnl9") pod "d3591ed5-dcce-46ac-b225-18116eeef308" (UID: "d3591ed5-dcce-46ac-b225-18116eeef308"). InnerVolumeSpecName "kube-api-access-nfnl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:05:45.360852 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.360829 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfnl9\" (UniqueName: \"kubernetes.io/projected/d3591ed5-dcce-46ac-b225-18116eeef308-kube-api-access-nfnl9\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:45.372588 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.372567 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-139754923509056"] Jul 15 11:05:45.372990 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.372976 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3591ed5-dcce-46ac-b225-18116eeef308" containerName="test-curl-7520-139754926687984" Jul 15 11:05:45.373071 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.372992 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3591ed5-dcce-46ac-b225-18116eeef308" containerName="test-curl-7520-139754926687984" Jul 15 11:05:45.373071 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.373066 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3591ed5-dcce-46ac-b225-18116eeef308" containerName="test-curl-7520-139754926687984" Jul 15 11:05:45.376043 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.376030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:45.389498 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.389479 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754923509056"] Jul 15 11:05:45.461536 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.461515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vfv8\" (UniqueName: \"kubernetes.io/projected/79e2c287-31d6-405c-86d9-b6a244184f01-kube-api-access-5vfv8\") pod \"test-curl-7520-139754923509056\" (UID: \"79e2c287-31d6-405c-86d9-b6a244184f01\") " pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:45.562860 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.562826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vfv8\" (UniqueName: \"kubernetes.io/projected/79e2c287-31d6-405c-86d9-b6a244184f01-kube-api-access-5vfv8\") pod \"test-curl-7520-139754923509056\" (UID: \"79e2c287-31d6-405c-86d9-b6a244184f01\") " pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:45.574220 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.574196 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vfv8\" (UniqueName: \"kubernetes.io/projected/79e2c287-31d6-405c-86d9-b6a244184f01-kube-api-access-5vfv8\") pod \"test-curl-7520-139754923509056\" (UID: \"79e2c287-31d6-405c-86d9-b6a244184f01\") " pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:45.684883 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.684829 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:45.817634 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:45.817594 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754923509056"] Jul 15 11:05:45.819888 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:05:45.819866 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79e2c287_31d6_405c_86d9_b6a244184f01.slice/crio-1fc60c0a67a16acd2e1f178aa83e4bdbf760cad386f667ef72ad359fdc975b87 WatchSource:0}: Error finding container 1fc60c0a67a16acd2e1f178aa83e4bdbf760cad386f667ef72ad359fdc975b87: Status 404 returned error can't find the container with id 1fc60c0a67a16acd2e1f178aa83e4bdbf760cad386f667ef72ad359fdc975b87 Jul 15 11:05:46.038201 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:46.038129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754923509056" event={"ID":"79e2c287-31d6-405c-86d9-b6a244184f01","Type":"ContainerStarted","Data":"1fc60c0a67a16acd2e1f178aa83e4bdbf760cad386f667ef72ad359fdc975b87"} Jul 15 11:05:46.039391 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:46.039370 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:46.039391 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:46.039389 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="835875ad285bec42c9d51f584a2c6243ae5f533315a2e88beeaf7ead8a81dacc" Jul 15 11:05:46.049093 ip-10-0-133-247 kubenswrapper[2579]: E0715 11:05:46.049068 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7520-139754926687984\" is forbidden: User \"system:node:ip-10-0-133-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-247.ec2.internal' and this object" podUID="d3591ed5-dcce-46ac-b225-18116eeef308" pod="opendatahub/test-curl-7520-139754926687984" Jul 15 11:05:46.351793 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:46.351766 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3591ed5-dcce-46ac-b225-18116eeef308" path="/var/lib/kubelet/pods/d3591ed5-dcce-46ac-b225-18116eeef308/volumes" Jul 15 11:05:47.046720 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:47.046680 2579 generic.go:356] "Generic (PLEG): container finished" podID="79e2c287-31d6-405c-86d9-b6a244184f01" containerID="8e3a700f1db037c89affde421a2d3e923556f109e32c60bf53ac65dfcd722075" exitCode=0 Jul 15 11:05:47.046902 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:47.046781 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754923509056" event={"ID":"79e2c287-31d6-405c-86d9-b6a244184f01","Type":"ContainerDied","Data":"8e3a700f1db037c89affde421a2d3e923556f109e32c60bf53ac65dfcd722075"} Jul 15 11:05:47.707342 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:47.707305 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" podUID="b987aa86-e276-47b9-be66-4e370d9a0ea9" containerName="main" containerID="cri-o://30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674" gracePeriod=45 Jul 15 11:05:47.951257 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:47.951235 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:05:48.052951 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.052863 2579 generic.go:356] "Generic (PLEG): container finished" podID="b987aa86-e276-47b9-be66-4e370d9a0ea9" containerID="30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674" exitCode=0 Jul 15 11:05:48.052951 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.052912 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" Jul 15 11:05:48.053122 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.052953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" event={"ID":"b987aa86-e276-47b9-be66-4e370d9a0ea9","Type":"ContainerDied","Data":"30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674"} Jul 15 11:05:48.053122 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.052997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68" event={"ID":"b987aa86-e276-47b9-be66-4e370d9a0ea9","Type":"ContainerDied","Data":"f23930fb6b2215fd57f994d1c00dd761e983581972e0ab71196cc5481823414a"} Jul 15 11:05:48.053122 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.053018 2579 scope.go:117] "RemoveContainer" containerID="30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674" Jul 15 11:05:48.061443 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.061414 2579 scope.go:117] "RemoveContainer" containerID="30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674" Jul 15 11:05:48.061703 ip-10-0-133-247 kubenswrapper[2579]: E0715 11:05:48.061684 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674\": container with ID starting with 30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674 not found: ID does not exist" containerID="30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674" Jul 15 11:05:48.061804 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.061713 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674"} err="failed to get container status \"30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674\": rpc error: code = NotFound desc = could not find container \"30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674\": container with ID starting with 30a09db35bb5586ed21c58fb35afce4b7582ae387c97a83f138806178e0c6674 not found: ID does not exist" Jul 15 11:05:48.087042 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.087014 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9fcc\" (UniqueName: \"kubernetes.io/projected/b987aa86-e276-47b9-be66-4e370d9a0ea9-kube-api-access-f9fcc\") pod \"b987aa86-e276-47b9-be66-4e370d9a0ea9\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " Jul 15 11:05:48.087180 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.087075 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-home\") pod \"b987aa86-e276-47b9-be66-4e370d9a0ea9\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " Jul 15 11:05:48.087180 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.087130 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-tmp-dir\") pod \"b987aa86-e276-47b9-be66-4e370d9a0ea9\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " Jul 15 11:05:48.087180 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.087174 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b987aa86-e276-47b9-be66-4e370d9a0ea9-tls-certs\") pod \"b987aa86-e276-47b9-be66-4e370d9a0ea9\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " Jul 15 11:05:48.087332 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.087221 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-dshm\") pod \"b987aa86-e276-47b9-be66-4e370d9a0ea9\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " Jul 15 11:05:48.087332 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.087264 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-model-cache\") pod \"b987aa86-e276-47b9-be66-4e370d9a0ea9\" (UID: \"b987aa86-e276-47b9-be66-4e370d9a0ea9\") " Jul 15 11:05:48.087439 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.087350 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-home" (OuterVolumeSpecName: "home") pod "b987aa86-e276-47b9-be66-4e370d9a0ea9" (UID: "b987aa86-e276-47b9-be66-4e370d9a0ea9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:05:48.087492 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.087470 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b987aa86-e276-47b9-be66-4e370d9a0ea9" (UID: "b987aa86-e276-47b9-be66-4e370d9a0ea9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:05:48.087646 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.087617 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-model-cache" (OuterVolumeSpecName: "model-cache") pod "b987aa86-e276-47b9-be66-4e370d9a0ea9" (UID: "b987aa86-e276-47b9-be66-4e370d9a0ea9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:05:48.088062 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.088042 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-home\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:48.088151 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.088067 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-tmp-dir\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:48.088151 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.088084 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-model-cache\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:48.089463 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.089430 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b987aa86-e276-47b9-be66-4e370d9a0ea9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b987aa86-e276-47b9-be66-4e370d9a0ea9" (UID: "b987aa86-e276-47b9-be66-4e370d9a0ea9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:05:48.089564 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.089479 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-dshm" (OuterVolumeSpecName: "dshm") pod "b987aa86-e276-47b9-be66-4e370d9a0ea9" (UID: "b987aa86-e276-47b9-be66-4e370d9a0ea9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 11:05:48.089564 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.089521 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b987aa86-e276-47b9-be66-4e370d9a0ea9-kube-api-access-f9fcc" (OuterVolumeSpecName: "kube-api-access-f9fcc") pod "b987aa86-e276-47b9-be66-4e370d9a0ea9" (UID: "b987aa86-e276-47b9-be66-4e370d9a0ea9"). InnerVolumeSpecName "kube-api-access-f9fcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:05:48.174708 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.174685 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:48.188838 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.188814 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b987aa86-e276-47b9-be66-4e370d9a0ea9-dshm\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:48.188838 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.188834 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9fcc\" (UniqueName: \"kubernetes.io/projected/b987aa86-e276-47b9-be66-4e370d9a0ea9-kube-api-access-f9fcc\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:48.188958 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.188844 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b987aa86-e276-47b9-be66-4e370d9a0ea9-tls-certs\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:48.204017 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.203996 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7520-139754923509056_79e2c287-31d6-405c-86d9-b6a244184f01/test-curl-7520-139754923509056/0.log" Jul 15 11:05:48.238709 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.238681 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-139754923509056"] Jul 15 11:05:48.246904 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.246878 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-139754923509056"] Jul 15 11:05:48.289598 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.289576 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vfv8\" (UniqueName: \"kubernetes.io/projected/79e2c287-31d6-405c-86d9-b6a244184f01-kube-api-access-5vfv8\") pod \"79e2c287-31d6-405c-86d9-b6a244184f01\" (UID: \"79e2c287-31d6-405c-86d9-b6a244184f01\") " Jul 15 11:05:48.291756 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.291721 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e2c287-31d6-405c-86d9-b6a244184f01-kube-api-access-5vfv8" (OuterVolumeSpecName: "kube-api-access-5vfv8") pod "79e2c287-31d6-405c-86d9-b6a244184f01" (UID: "79e2c287-31d6-405c-86d9-b6a244184f01"). InnerVolumeSpecName "kube-api-access-5vfv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:05:48.351775 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.351731 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e2c287-31d6-405c-86d9-b6a244184f01" path="/var/lib/kubelet/pods/79e2c287-31d6-405c-86d9-b6a244184f01/volumes" Jul 15 11:05:48.370102 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.370079 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-139754923509056"] Jul 15 11:05:48.370454 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.370438 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b987aa86-e276-47b9-be66-4e370d9a0ea9" containerName="main" Jul 15 11:05:48.370538 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.370456 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b987aa86-e276-47b9-be66-4e370d9a0ea9" containerName="main" Jul 15 11:05:48.370538 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.370480 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e2c287-31d6-405c-86d9-b6a244184f01" containerName="test-curl-7520-139754923509056" Jul 15 11:05:48.370538 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.370489 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e2c287-31d6-405c-86d9-b6a244184f01" containerName="test-curl-7520-139754923509056" Jul 15 11:05:48.370682 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.370576 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e2c287-31d6-405c-86d9-b6a244184f01" containerName="test-curl-7520-139754923509056" Jul 15 11:05:48.370682 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.370593 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b987aa86-e276-47b9-be66-4e370d9a0ea9" containerName="main" Jul 15 11:05:48.373622 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.373603 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:48.380247 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.380229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754923509056"] Jul 15 11:05:48.383925 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.383906 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68"] Jul 15 11:05:48.386783 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.386763 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ceb9f0/rate-test-model-ceb9f0-kserve-6d959fb755-wnl68"] Jul 15 11:05:48.390258 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.390242 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vfv8\" (UniqueName: \"kubernetes.io/projected/79e2c287-31d6-405c-86d9-b6a244184f01-kube-api-access-5vfv8\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:48.491250 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.491222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckjdh\" (UniqueName: \"kubernetes.io/projected/49051fa8-ebc1-47d7-acf2-ca939f3a54b4-kube-api-access-ckjdh\") pod \"test-curl-7520-139754923509056\" (UID: \"49051fa8-ebc1-47d7-acf2-ca939f3a54b4\") " pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:48.592122 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.592091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckjdh\" (UniqueName: \"kubernetes.io/projected/49051fa8-ebc1-47d7-acf2-ca939f3a54b4-kube-api-access-ckjdh\") pod \"test-curl-7520-139754923509056\" (UID: \"49051fa8-ebc1-47d7-acf2-ca939f3a54b4\") " pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:48.602161 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.602095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckjdh\" (UniqueName: \"kubernetes.io/projected/49051fa8-ebc1-47d7-acf2-ca939f3a54b4-kube-api-access-ckjdh\") pod \"test-curl-7520-139754923509056\" (UID: \"49051fa8-ebc1-47d7-acf2-ca939f3a54b4\") " pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:48.682853 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:48.682826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:49.029309 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:49.028968 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-139754923509056"] Jul 15 11:05:49.031773 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:05:49.031731 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49051fa8_ebc1_47d7_acf2_ca939f3a54b4.slice/crio-3137f91c49df16455218e5921662a090c0336bbe9d01a74bf19852f9b6d51b06 WatchSource:0}: Error finding container 3137f91c49df16455218e5921662a090c0336bbe9d01a74bf19852f9b6d51b06: Status 404 returned error can't find the container with id 3137f91c49df16455218e5921662a090c0336bbe9d01a74bf19852f9b6d51b06 Jul 15 11:05:49.060846 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:49.060823 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:49.060956 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:49.060833 2579 scope.go:117] "RemoveContainer" containerID="8e3a700f1db037c89affde421a2d3e923556f109e32c60bf53ac65dfcd722075" Jul 15 11:05:49.061958 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:49.061940 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754923509056" event={"ID":"49051fa8-ebc1-47d7-acf2-ca939f3a54b4","Type":"ContainerStarted","Data":"3137f91c49df16455218e5921662a090c0336bbe9d01a74bf19852f9b6d51b06"} Jul 15 11:05:50.070431 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:50.070390 2579 generic.go:356] "Generic (PLEG): container finished" podID="49051fa8-ebc1-47d7-acf2-ca939f3a54b4" containerID="ecf163011d575659ee88198785bb07a790b079e41bd7bea8e2c0946ab18b11a1" exitCode=0 Jul 15 11:05:50.070830 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:50.070431 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-139754923509056" event={"ID":"49051fa8-ebc1-47d7-acf2-ca939f3a54b4","Type":"ContainerDied","Data":"ecf163011d575659ee88198785bb07a790b079e41bd7bea8e2c0946ab18b11a1"} Jul 15 11:05:50.351534 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:50.351504 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b987aa86-e276-47b9-be66-4e370d9a0ea9" path="/var/lib/kubelet/pods/b987aa86-e276-47b9-be66-4e370d9a0ea9/volumes" Jul 15 11:05:51.212522 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.212498 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:51.237783 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.237762 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7520-139754923509056_49051fa8-ebc1-47d7-acf2-ca939f3a54b4/test-curl-7520-139754923509056/0.log" Jul 15 11:05:51.267157 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.267127 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-139754923509056"] Jul 15 11:05:51.273750 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.273715 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-139754923509056"] Jul 15 11:05:51.316609 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.316586 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckjdh\" (UniqueName: \"kubernetes.io/projected/49051fa8-ebc1-47d7-acf2-ca939f3a54b4-kube-api-access-ckjdh\") pod \"49051fa8-ebc1-47d7-acf2-ca939f3a54b4\" (UID: \"49051fa8-ebc1-47d7-acf2-ca939f3a54b4\") " Jul 15 11:05:51.318841 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.318813 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49051fa8-ebc1-47d7-acf2-ca939f3a54b4-kube-api-access-ckjdh" (OuterVolumeSpecName: "kube-api-access-ckjdh") pod "49051fa8-ebc1-47d7-acf2-ca939f3a54b4" (UID: "49051fa8-ebc1-47d7-acf2-ca939f3a54b4"). InnerVolumeSpecName "kube-api-access-ckjdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 11:05:51.417921 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.417881 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckjdh\" (UniqueName: \"kubernetes.io/projected/49051fa8-ebc1-47d7-acf2-ca939f3a54b4-kube-api-access-ckjdh\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:51.465490 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.465456 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6"] Jul 15 11:05:51.465815 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.465802 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49051fa8-ebc1-47d7-acf2-ca939f3a54b4" containerName="test-curl-7520-139754923509056" Jul 15 11:05:51.465861 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.465817 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="49051fa8-ebc1-47d7-acf2-ca939f3a54b4" containerName="test-curl-7520-139754923509056" Jul 15 11:05:51.465908 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.465880 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="49051fa8-ebc1-47d7-acf2-ca939f3a54b4" containerName="test-curl-7520-139754923509056" Jul 15 11:05:51.468885 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.468867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6" Jul 15 11:05:51.471379 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.471362 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zj826\"" Jul 15 11:05:51.471484 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.471383 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 11:05:51.478197 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.478168 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6"] Jul 15 11:05:51.519363 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.519332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf518df-8fbf-47e0-981f-3497519c71dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6\" (UID: \"fbf518df-8fbf-47e0-981f-3497519c71dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6" Jul 15 11:05:51.620698 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.620667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf518df-8fbf-47e0-981f-3497519c71dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6\" (UID: \"fbf518df-8fbf-47e0-981f-3497519c71dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6" Jul 15 11:05:51.621258 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.621231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf518df-8fbf-47e0-981f-3497519c71dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6\" (UID: \"fbf518df-8fbf-47e0-981f-3497519c71dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6" Jul 15 11:05:51.780131 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.780049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6" Jul 15 11:05:51.910416 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:51.910380 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6"] Jul 15 11:05:51.914431 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:05:51.914406 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf518df_8fbf_47e0_981f_3497519c71dd.slice/crio-e9e845fa88adb655785b6f223f5225964f5a79fc4d46a8e29bcff0036f496d8e WatchSource:0}: Error finding container e9e845fa88adb655785b6f223f5225964f5a79fc4d46a8e29bcff0036f496d8e: Status 404 returned error can't find the container with id e9e845fa88adb655785b6f223f5225964f5a79fc4d46a8e29bcff0036f496d8e Jul 15 11:05:52.079193 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:52.079100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:52.082413 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:52.082389 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3137f91c49df16455218e5921662a090c0336bbe9d01a74bf19852f9b6d51b06" Jul 15 11:05:52.082571 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:52.082417 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:52.083589 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:52.083565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6" event={"ID":"fbf518df-8fbf-47e0-981f-3497519c71dd","Type":"ContainerStarted","Data":"e9e845fa88adb655785b6f223f5225964f5a79fc4d46a8e29bcff0036f496d8e"} Jul 15 11:05:52.100267 ip-10-0-133-247 kubenswrapper[2579]: E0715 11:05:52.100243 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7520-139754923509056\" is forbidden: User \"system:node:ip-10-0-133-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-247.ec2.internal' and this object" podUID="49051fa8-ebc1-47d7-acf2-ca939f3a54b4" pod="opendatahub/test-curl-7520-139754923509056" Jul 15 11:05:52.179605 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:52.179576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:52.351170 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:52.351127 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49051fa8-ebc1-47d7-acf2-ca939f3a54b4" path="/var/lib/kubelet/pods/49051fa8-ebc1-47d7-acf2-ca939f3a54b4/volumes" Jul 15 11:05:53.090162 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:53.090118 2579 generic.go:356] "Generic (PLEG): container finished" podID="fbf518df-8fbf-47e0-981f-3497519c71dd" containerID="86d575ecfd8915b82922862a25e42d87a2abd27eaf273c24e8d1e16dbe084506" exitCode=0 Jul 15 11:05:53.090373 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:53.090197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6" event={"ID":"fbf518df-8fbf-47e0-981f-3497519c71dd","Type":"ContainerDied","Data":"86d575ecfd8915b82922862a25e42d87a2abd27eaf273c24e8d1e16dbe084506"} Jul 15 11:05:54.234152 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:54.234122 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6" Jul 15 11:05:54.344222 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:54.344181 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf518df-8fbf-47e0-981f-3497519c71dd-maas-api-service-ca\") pod \"fbf518df-8fbf-47e0-981f-3497519c71dd\" (UID: \"fbf518df-8fbf-47e0-981f-3497519c71dd\") " Jul 15 11:05:54.344563 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:54.344542 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf518df-8fbf-47e0-981f-3497519c71dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbf518df-8fbf-47e0-981f-3497519c71dd" (UID: "fbf518df-8fbf-47e0-981f-3497519c71dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:05:54.445831 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:54.445793 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf518df-8fbf-47e0-981f-3497519c71dd-maas-api-service-ca\") on node \"ip-10-0-133-247.ec2.internal\" DevicePath \"\"" Jul 15 11:05:55.103087 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:55.103052 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6" Jul 15 11:05:55.103344 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:55.103081 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6" event={"ID":"fbf518df-8fbf-47e0-981f-3497519c71dd","Type":"ContainerDied","Data":"e9e845fa88adb655785b6f223f5225964f5a79fc4d46a8e29bcff0036f496d8e"} Jul 15 11:05:55.103344 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:05:55.103129 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e845fa88adb655785b6f223f5225964f5a79fc4d46a8e29bcff0036f496d8e" Jul 15 11:06:02.130259 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:02.130217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:06.631248 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:06.631209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:13.953133 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:13.953048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:14.031335 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:14.031304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:17.903310 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:17.903274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:18.003997 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:18.003966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:18.437753 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:18.437718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:18.539178 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:18.539091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:28.634044 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:28.634010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:32.886723 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:32.886692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:40.012318 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:40.012279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:40.122760 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:40.122701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:43.617708 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:43.617674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:43.843022 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:43.842988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:44.267154 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:44.267122 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bc45ddbc6-2hlk6_cb119164-c78d-4fc7-a093-bf9508685d95/authorino/0.log" Jul 15 11:06:50.504429 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:50.504396 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-656947bb4-44wmg_3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1/copy-manifests/0.log" Jul 15 11:06:50.518380 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:50.518352 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-656947bb4-44wmg_3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1/manager/0.log" Jul 15 11:06:51.302177 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:51.302141 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735190-twfmx_39d06b3e-cbba-4b70-b396-50d5e3208576/cleanup/0.log" Jul 15 11:06:52.225949 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:52.225913 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-97bab6-984dw_cfde8a05-5d88-4804-be00-2d5f0e6399e5/revoke-keys/0.log" Jul 15 11:06:52.339867 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:52.339826 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-a47e9a-7tvl7_8a313961-08ba-4d3a-8ad4-6d25c32bcfeb/revoke-keys/0.log" Jul 15 11:06:52.800966 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:52.800934 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-b2a040-nzbg6_fbf518df-8fbf-47e0-981f-3497519c71dd/revoke-keys/0.log" Jul 15 11:06:54.430074 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:54.430041 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bc45ddbc6-2hlk6_cb119164-c78d-4fc7-a093-bf9508685d95/authorino/0.log" Jul 15 11:06:55.634857 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:55.634826 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb67897rvc_95f1570a-64b2-4e3b-85d2-5470f77ba856/istio-proxy/0.log" Jul 15 11:06:55.872490 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:55.872461 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-544bf6b555-8zcfv_88a54609-e5a2-47ed-b449-1bb7588e6bc3/kube-auth-proxy/0.log" Jul 15 11:06:56.947352 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:56.947320 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-w5trg_c8c8e7f3-b8b5-49f6-bb0f-7660ac5fc33d/main/0.log" Jul 15 11:06:57.309174 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:57.309097 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-gjm86_427624df-65e0-404c-9fd5-72d2ca92c989/main/0.log" Jul 15 11:06:57.426419 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:06:57.426395 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-f699p_df7cc2c0-554d-4cb6-a830-27d7d0282252/main/0.log" Jul 15 11:07:02.967036 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:02.966991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:03.072021 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:03.071989 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:06.332109 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:06.332078 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mw4m6_4fd5ac79-bd46-4d5e-9c77-9df73609b5b8/global-pull-secret-syncer/0.log" Jul 15 11:07:06.391835 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:06.391804 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9dvfc_3e068402-7f79-4214-b488-5ee985a633ec/konnectivity-agent/0.log" Jul 15 11:07:06.540153 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:06.540116 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-247.ec2.internal_740d9dfc5ebbe2db89963886b2766aa3/haproxy/0.log" Jul 15 11:07:12.223585 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:12.223549 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bc45ddbc6-2hlk6_cb119164-c78d-4fc7-a093-bf9508685d95/authorino/0.log" Jul 15 11:07:14.314510 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:14.314480 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78ca7ebb-b1b6-4b52-ba61-860ac7c46c74/alertmanager/0.log" Jul 15 11:07:14.369784 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:14.369736 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78ca7ebb-b1b6-4b52-ba61-860ac7c46c74/config-reloader/0.log" Jul 15 11:07:14.417698 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:14.417671 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78ca7ebb-b1b6-4b52-ba61-860ac7c46c74/kube-rbac-proxy-web/0.log" Jul 15 11:07:14.477051 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:14.477022 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78ca7ebb-b1b6-4b52-ba61-860ac7c46c74/kube-rbac-proxy/0.log" Jul 15 11:07:14.572732 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:14.572655 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78ca7ebb-b1b6-4b52-ba61-860ac7c46c74/kube-rbac-proxy-metric/0.log" Jul 15 11:07:14.651883 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:14.651855 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78ca7ebb-b1b6-4b52-ba61-860ac7c46c74/prom-label-proxy/0.log" Jul 15 11:07:14.718911 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:14.718842 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78ca7ebb-b1b6-4b52-ba61-860ac7c46c74/init-config-reloader/0.log" Jul 15 11:07:14.815973 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:14.815941 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-9cqph_ceba8a0a-f212-4835-9f43-32e037872cde/kube-state-metrics/0.log" Jul 15 11:07:14.862671 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:14.862638 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-9cqph_ceba8a0a-f212-4835-9f43-32e037872cde/kube-rbac-proxy-main/0.log" Jul 15 11:07:14.903531 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:14.903488 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-9cqph_ceba8a0a-f212-4835-9f43-32e037872cde/kube-rbac-proxy-self/0.log" Jul 15 11:07:15.139463 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.139384 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74flj_ba908431-954b-413f-a500-36ea818f03d8/node-exporter/0.log" Jul 15 11:07:15.167339 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.167312 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74flj_ba908431-954b-413f-a500-36ea818f03d8/kube-rbac-proxy/0.log" Jul 15 11:07:15.198290 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.198263 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74flj_ba908431-954b-413f-a500-36ea818f03d8/init-textfile/0.log" Jul 15 11:07:15.849558 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.849521 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr"] Jul 15 11:07:15.850006 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.849885 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf518df-8fbf-47e0-981f-3497519c71dd" containerName="revoke-keys" Jul 15 11:07:15.850006 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.849896 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf518df-8fbf-47e0-981f-3497519c71dd" containerName="revoke-keys" Jul 15 11:07:15.850006 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.849963 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbf518df-8fbf-47e0-981f-3497519c71dd" containerName="revoke-keys" Jul 15 11:07:15.853170 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.853151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:15.857403 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.857376 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68cgz\"/\"openshift-service-ca.crt\"" Jul 15 11:07:15.857952 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.857927 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68cgz\"/\"kube-root-ca.crt\"" Jul 15 11:07:15.858591 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.858574 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-68cgz\"/\"default-dockercfg-8vwgn\"" Jul 15 11:07:15.860962 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.860943 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b98d786b-52mzv_66bb1ec0-627c-495a-9211-46eb2c1a8e0d/thanos-query/0.log" Jul 15 11:07:15.865926 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.865906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr"] Jul 15 11:07:15.908435 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.908404 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b98d786b-52mzv_66bb1ec0-627c-495a-9211-46eb2c1a8e0d/kube-rbac-proxy-web/0.log" Jul 15 11:07:15.937100 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.937064 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b98d786b-52mzv_66bb1ec0-627c-495a-9211-46eb2c1a8e0d/kube-rbac-proxy/0.log" Jul 15 11:07:15.964252 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.964220 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b98d786b-52mzv_66bb1ec0-627c-495a-9211-46eb2c1a8e0d/prom-label-proxy/0.log" Jul 15 11:07:15.993067 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:15.993039 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b98d786b-52mzv_66bb1ec0-627c-495a-9211-46eb2c1a8e0d/kube-rbac-proxy-rules/0.log" Jul 15 11:07:16.020703 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.020671 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b98d786b-52mzv_66bb1ec0-627c-495a-9211-46eb2c1a8e0d/kube-rbac-proxy-metrics/0.log" Jul 15 11:07:16.041993 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.041956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-sys\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.042151 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.042000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-proc\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.042151 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.042033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-lib-modules\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.042151 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.042093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lll45\" (UniqueName: \"kubernetes.io/projected/91d1e16e-62d5-4dd7-954c-7d782a12ba38-kube-api-access-lll45\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.042151 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.042117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-podres\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.143407 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.143319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lll45\" (UniqueName: \"kubernetes.io/projected/91d1e16e-62d5-4dd7-954c-7d782a12ba38-kube-api-access-lll45\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.143407 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.143354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-podres\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.143407 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.143405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-sys\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.143693 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.143429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-proc\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.143693 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.143454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-lib-modules\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.143693 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.143530 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-sys\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.143693 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.143537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-proc\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.143693 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.143602 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-lib-modules\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.143693 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.143619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91d1e16e-62d5-4dd7-954c-7d782a12ba38-podres\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.154396 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.154371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lll45\" (UniqueName: \"kubernetes.io/projected/91d1e16e-62d5-4dd7-954c-7d782a12ba38-kube-api-access-lll45\") pod \"perf-node-gather-daemonset-xt8vr\" (UID: \"91d1e16e-62d5-4dd7-954c-7d782a12ba38\") " pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.164412 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.164386 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.309541 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.309388 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr"] Jul 15 11:07:16.312452 ip-10-0-133-247 kubenswrapper[2579]: W0715 11:07:16.312421 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod91d1e16e_62d5_4dd7_954c_7d782a12ba38.slice/crio-00b6927a5d59506ca30ff6fd0e0234f9eefb1f114b4670033f5d9430383da711 WatchSource:0}: Error finding container 00b6927a5d59506ca30ff6fd0e0234f9eefb1f114b4670033f5d9430383da711: Status 404 returned error can't find the container with id 00b6927a5d59506ca30ff6fd0e0234f9eefb1f114b4670033f5d9430383da711 Jul 15 11:07:16.522848 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.522799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" event={"ID":"91d1e16e-62d5-4dd7-954c-7d782a12ba38","Type":"ContainerStarted","Data":"a701bc36389b90f14a01593a801ef2e112fee5db54f53177296a8492a79c8147"} Jul 15 11:07:16.523044 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.522890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" event={"ID":"91d1e16e-62d5-4dd7-954c-7d782a12ba38","Type":"ContainerStarted","Data":"00b6927a5d59506ca30ff6fd0e0234f9eefb1f114b4670033f5d9430383da711"} Jul 15 11:07:16.523044 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.522917 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:16.543393 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:16.543337 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" podStartSLOduration=1.5433198689999998 podStartE2EDuration="1.543319869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:07:16.542196014 +0000 UTC m=+2838.675389350" watchObservedRunningTime="2026-07-15 11:07:16.543319869 +0000 UTC m=+2838.676513241" Jul 15 11:07:17.124356 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:17.124329 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-r2s6b_d775f48e-8d09-4df4-818f-29209bcafcac/networking-console-plugin/0.log" Jul 15 11:07:20.110508 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:20.110478 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-86hn4_86a3decd-de00-4e20-bb88-e5bde87107be/dns/0.log" Jul 15 11:07:20.139423 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:20.139404 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-86hn4_86a3decd-de00-4e20-bb88-e5bde87107be/kube-rbac-proxy/0.log" Jul 15 11:07:20.257316 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:20.257289 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q95hm_e0471e5a-45bf-4183-b286-8904049cb8ce/dns-node-resolver/0.log" Jul 15 11:07:20.957670 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:20.957641 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zn4vg_cf65ce46-ab2d-4f15-b36f-aa9dd8fbfca2/node-ca/0.log" Jul 15 11:07:21.995285 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:21.995256 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb67897rvc_95f1570a-64b2-4e3b-85d2-5470f77ba856/istio-proxy/0.log" Jul 15 11:07:22.168088 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:22.168062 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-544bf6b555-8zcfv_88a54609-e5a2-47ed-b449-1bb7588e6bc3/kube-auth-proxy/0.log" Jul 15 11:07:22.538354 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:22.538328 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-68cgz/perf-node-gather-daemonset-xt8vr" Jul 15 11:07:23.141798 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:23.141764 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6snt9_2ca3ff4f-3931-43fa-ad4d-cf88057f88f3/serve-healthcheck-canary/0.log" Jul 15 11:07:23.952720 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:23.952694 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sprhs_f2fcae6d-fd94-42d8-82fd-606fa907ca19/kube-rbac-proxy/0.log" Jul 15 11:07:23.978500 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:23.978477 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sprhs_f2fcae6d-fd94-42d8-82fd-606fa907ca19/exporter/0.log" Jul 15 11:07:24.012812 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:24.012784 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sprhs_f2fcae6d-fd94-42d8-82fd-606fa907ca19/extractor/0.log" Jul 15 11:07:26.114297 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:26.114270 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb67897rvc_95f1570a-64b2-4e3b-85d2-5470f77ba856/istio-proxy/0.log" Jul 15 11:07:26.279968 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:26.279903 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-544bf6b555-8zcfv_88a54609-e5a2-47ed-b449-1bb7588e6bc3/kube-auth-proxy/0.log" Jul 15 11:07:26.883250 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:26.883218 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-656947bb4-44wmg_3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1/manager/0.log" Jul 15 11:07:26.908179 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:26.908153 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-656947bb4-44wmg_3a0f2f2b-1d6a-4723-9f23-3495ed57a9b1/copy-manifests/0.log" Jul 15 11:07:27.689410 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:27.689386 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb67897rvc_95f1570a-64b2-4e3b-85d2-5470f77ba856/istio-proxy/0.log" Jul 15 11:07:27.851553 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:27.851520 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-544bf6b555-8zcfv_88a54609-e5a2-47ed-b449-1bb7588e6bc3/kube-auth-proxy/0.log" Jul 15 11:07:28.435772 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:28.435689 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-587d8d67f-6t8rd_61806022-8dd9-48d9-bd55-8aaa6b5a9f4a/manager/0.log" Jul 15 11:07:28.501710 ip-10-0-133-247 kubenswrapper[2579]: I0715 11:07:28.501679 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-6qxv2_1d55a01d-6152-4878-8db5-082e0970b2e4/openshift-lws-operator/0.log"